search

Azure / azure-webjobs-sdk

Azure WebJobs SDK
MIT License
739 stars 358 forks source link

Azure WebJob Host/Runtime doesn't support Managed Identity (using the Azure WebJob SDK 3.x) #3069

Open gautammoulik opened 7 months ago

gautammoulik commented 7 months ago

Azure WebJob runs 2 types of code - a) the WebJob host/Runtime and b) Application code (write by the developers) - For the second part it's very easy to connect to a Azure Storage using a Managed Identity. However, for the first part which is WebJob Host - it communicate with the Azure Storage for various internal reasons, any Storage related communication details are controlled thru configuration like - "AzureWebJobStorage" and "AzureWebJobsDashboard". None of these configuration setting allow to configure a Managed Identity, neither there is any documentation about the alternative configurations can be used to force the WebJob host/Runtime use Managed Identity while communicating with the Azure Storage.

Repro steps

Provide the steps required to reproduce the problem

  1. Step A Create a WebJob with a timer triggered function. Remove the AzureWebJobStorage configuration (or keep the value as "" ).

Note - Although some related documentation about Azure Function claims about Azure Function works with the below configurations - AzureWebJobsStorageaccountName AzureWebJobsStorageCredential

Adding the above settings didn't do any help.

  1. Step B

Expected behavior

The WebJob function should be running when the timer triggered( or the configured time expired).

Actual behavior

The WebJob throws an error on the Timer expired event - Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException: The listener for function 'TokenIngestAgentQueueFunction.ProcessIngestJobRequestFromQueueAsync' was unable to start. ---> System.ArgumentNullException: Value cannot be null. Parameter name: connectionString at Microsoft.Azure.Storage.CloudStorageAccount.Parse(String connectionString) at Microsoft.Azure.WebJobs.Extensions.Timers.StorageScheduleMonitor.get_TimerStatusDirectory() in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Scheduling\StorageScheduleMonitor.cs:line 86 at Microsoft.Azure.WebJobs.Extensions.Timers.StorageScheduleMonitor.GetStatusBlobReference(String timerName) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Scheduling\StorageScheduleMonitor.cs:line 144 at Microsoft.Azure.WebJobs.Extensions.Timers.StorageScheduleMonitor.d10.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Extensions.Timers.Listeners.TimerListener.d27.MoveNext() in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Listener\TimerListener.cs:line 99 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener.d13.MoveNext() in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonListener.cs:line 72 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.d13.MoveNext() in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 69 --- End of inner exception stack trace --- at Microsoft.Azure.WebJobs.Host.RecoverableException.TryRecover(ILogger logger) in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Exceptions\RecoverableException.cs:line 81 at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.d13.MoveNext() in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 81 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.d12.MoveNext() in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 61 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener.d4.MoveNext() in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\CompositeListener.cs:line 39 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener.d8.MoveNext() in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\ListenerFactoryListener.cs:line 47 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener.d5.MoveNext() in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\ShutdownListener.cs:line 29 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.JobHost.d23.MoveNext() in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\JobHost.cs:line 101 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.Extensions.Hosting.Internal.Host.d9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.d4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host) at Microsoft.SupplyChain.TokenIngest.Agent.Program.Main() in C:__w\1\s\src\TokenIngestAgent\Program.cs:line 142

Known workarounds

N/A

Related information

Provide any related information

mattchenderson commented 7 months ago

Identity is only supported starting with version 5.x of the storage extension. You can see in the stack trace that Microsoft.Azure.Storage is being used, which is the older library. You instead want to be using the newer Azure.Storage packages. This transition is the key difference between 4.x and 5.x, and support for identity is one of the main features this enables.

When navigating the difference in the types used, the Storage SDK team has provided the following transition guide, which may help: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/AzureStorageNetMigrationV12.md

gautammoulik commented 6 months ago

@mattchenderson - Thanks for your response. To be clear - the business code (aka the code I wrote to run inside the WebJob) can connect to the Azure Storage using the Managed Identity without any issues. I have explicitly added a reference to Azure.Storage.Blob package only.

The issue is related to the WebJob infra code (which comes along with the Microsoft.Azure.WebJobs and it's extension packages ), for example a Timer trigger based Web job needs to communicate with Storage Account. It seems the WebJob infra doesn't know/support how to connect to a Storage using a Managed Identity.

I am using the below WebJob packages - Microsoft.Azure.WebJobs - 3.0.39 Microsoft.Azure.WebJobs.Extensions - 4.0.1 Microsoft.Azure.WebJobs.Extensions.Storage - 4.0.2

Can you please share me a sample project which runs a WebJob (with timer trigger) and the WebJob infra code connecting to Storage account (to acquire a Lock while starting) using the Managed Identity.

Thanks !! Gautam

eureka-gh commented 6 months ago

I am having the same issue in my webjob program where timer trigger requires StorageAccount connection string in AzureWebJobsStorage. According to microsoft learn: https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference?tabs=blob&pivots=programming-language-csharp#connecting-to-host-storage-with-an-identity

The Azure Functions host uses the storage connection set in AzureWebJobsStorage to enable core behaviors such as coordinating singleton execution of timer triggers and default app key storage. This connection can also be configured to use an identity.

The solution is to use "AzureWebJobsStorage__accountName" with proper roles (AKA, RBAC).

It may work for FunctionApp, but not for WebJob. I did the same setting in my WebJob project (Microsoft.Azure.WebJobs" Version="3.0.33"), it doesn't work as @gautammoulik mentioned.

Also, according to @mattchenderson, I checked the stack track below, it's Microsoft.Azure.Storage module who thrown this exception. But how I can replace Microsoft.Azure.Storage nuget package into Azure.Storage in a webjob project?

The listener for function 'CustomSchedulerFunction.CustomSchedulerHandler' was unable to start. Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException: The listener for function 'CustomSchedulerFunction.CustomSchedulerHandler' was unable to start. ---> System.ArgumentNullException: Value cannot be null. (Parameter 'connectionString') at Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse(String connectionString) at Microsoft.Azure.WebJobs.Extensions.Timers.StorageScheduleMonitor.get_TimerStatusDirectory() in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Scheduling\StorageScheduleMonitor.cs:line 77 at Microsoft.Azure.WebJobs.Extensions.Timers.StorageScheduleMonitor.GetStatusBlobReference(String timerName) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Scheduling\StorageScheduleMonitor.cs:line 144 at Microsoft.Azure.WebJobs.Extensions.Timers.StorageScheduleMonitor.GetStatusAsync(String timerName) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Scheduling\StorageScheduleMonitor.cs:line 93 at Microsoft.Azure.WebJobs.Extensions.Timers.Listeners.TimerListener.StartAsync(CancellationToken cancellationToken) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Listener\TimerListener.cs:line 99 at Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener.StartAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonListener.cs:line 72 at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(CancellationToken cancellationToken, Boolean allowRetry) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 69 --- End of inner exception stack trace ---

I've tried to add "Microsoft.Azure.WebJobs.Extensions.Storage" 4.0.5 in my WebJob, (the 5.x.x doen't compatible with webjob 3.x.x) I added .AddAzureStorage() builder.ConfigureWebJobs(webJobBuilder => { webJobBuilder.AddAzureStorageCoreServices(); webJobBuilder.AddAzureStorage(); webJobBuilder.AddServiceBus(options =>{}) // ... }

eureka-gh commented 6 months ago

I am having the same issue in my webjob program where timer trigger requires StorageAccount connection string in AzureWebJobsStorage which I want to get rid of with MSI.

According to this microsoft learn link:

The Azure Functions host uses the storage connection set in AzureWebJobsStorage to enable core behaviors such as coordinating singleton execution of timer triggers and default app key storage. This connection can also be configured to use an identity. The solution is to use "AzureWebJobsStorage__accountName" with proper roles (AKA, RBAC).

I did the same setting in my WebJob project (Microsoft.Azure.WebJobs" Version="3.0.33", "Microsoft.Azure.WebJobs.Extensions" Version="3.0.6"), it doesn't work as @gautammoulik mentioned. The error message below is to do with null connection string (ALthough, I've set Environment.SetEnvironmentVariable("AzureWebJobsStorage__accountName", storageAcctOptions.AccountName) in Program.cs/GetJobHost() )

The listener for function 'CustomSchedulerFunction.CustomSchedulerHandler' was unable to start.
Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException: The listener for function 'CustomSchedulerFunction.CustomSchedulerHandler' was unable to start.
---> System.ArgumentNullException: Value cannot be null. (Parameter 'connectionString')
at Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse(String connectionString)
at Microsoft.Azure.WebJobs.Extensions.Timers.StorageScheduleMonitor.get_TimerStatusDirectory() in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Scheduling\StorageScheduleMonitor.cs:line 77
at Microsoft.Azure.WebJobs.Extensions.Timers.StorageScheduleMonitor.GetStatusBlobReference(String timerName) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Scheduling\StorageScheduleMonitor.cs:line 144
at Microsoft.Azure.WebJobs.Extensions.Timers.StorageScheduleMonitor.GetStatusAsync(String timerName) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Scheduling\StorageScheduleMonitor.cs:line 93
at Microsoft.Azure.WebJobs.Extensions.Timers.Listeners.TimerListener.StartAsync(CancellationToken cancellationToken) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Listener\TimerListener.cs:line 99
at Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener.StartAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonListener.cs:line 72
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(CancellationToken cancellationToken, Boolean allowRetry) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 69
--- End of inner exception stack trace ---

@gautammoulik, Luckly, after a couple of attempts, it turns out the "AzureWebJobsStorage__accountName" cannot be interpreted by WebJobs.Extension <= 3.0.33. If I upgrade the Microsoft.Azure.WebJobs.Extensions to "5.0.0" as of 05/12/2024. ("Microsoft.Azure.WebJobs" to 3.0.39 seems an optional), MSI works for me.

Although, I still don't know how to use a user managed identity or AAD App identity. I am guessing you need to Add Microsoft.Azure.WebJobs.Extensions.Timers.Storage, and then update TimerOptions like: webJobBuilder.AddTimers(/* options => TimerOptions.AddBlobServiceClient() */).

I don't have time to test, but I feel it's in the right track though. Hope it helps.

PS. my project is based on .NetCore 3.1.