After migration to .NET 8(Isolated) from .NET 6(In-Proc), Azure Isolated function (queue triggered) throwing timeout issue.

[abkdas123wk]

The Azure function (queue triggered) is hosted in AKS cluster (Linux Debian). We are migrating from .NET 6 to .NET 8, in this process we migrated this in-proc azure function to isolated function. After migration we started receiving timeout exception whenever we submit 20+ messages to the queue quickly. In .NET 6 (In-Proc), we never faced this issue. This queue triggered isolated function makes some http calls, DB calls, does some processing and generates pdf. These are the settings and packages in .csproj file: PropertyGroup TargetFramework net8.0 AzureFunctionsVersion v4 LangVersion 12.0 OutputType Exe PropertyGroup ItemGroup PackageReference Include="Azure.Identity" Version="1.11.2" PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" PackageReference Include="Microsoft.Azure.Core.NewtonsoftJson" Version="2.0.0" PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.3-preview1" PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Blobs" Version="6.4.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.4.0" PackageReference Include="Microsoft.Extensions.Http.Polly" Version="8.0.4" PackageReference Include="Microsoft.Data.SqlClient" Version="5.2.0" PackageReference Include="Microsoft.Extensions.Logging" Version="8.0.0" PackageReference Include="Microsoft.CodeAnalysis.Common" Version="4.9.2" PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.9.2" PackageReference Include="NuGet.Packaging" Version="6.9.1" PackageReference Include="RazorEngineCore" Version="2024.4.1" PackageReference Include="System.Security.Cryptography.Xml" Version="8.0.0" PackageReference Include="Microsoft.EntityFrameworkCore" Version="8.0.4" PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.4" ItemGroup

Program.cs public class Program { public IConfiguration Configuration { get; private set; } static async Task Main(string[] args) { await CreateHostBuilder(args).Build().RunAsync(); }

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureFunctionsWorkerDefaults(builder =>
        {
            WorkerConfigurationExtensions.UseNewtonsoftJson(builder);
        }, options =>
        {
            options.EnableUserCodeException = true;
        })
        .ConfigureAppConfiguration((hostingContext, config) =>
        {
            var envName = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production";
            var basePath = Directory.GetCurrentDirectory();
            config.SetBasePath(basePath);
            config.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                                    .AddJsonFile($"appsettings.{envName}.json", true, true)
                                    .AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("AxcessEnv")}.json", true, true)
                                    .AddEnvironmentVariables()
                                    .TryAddAzureKeyVault();

            config.Build();
        })
        .ConfigureServices((hostContext, services) =>
        {
            services.Configure<WorkerOptions>(opt =>
            {
                opt.EnableUserCodeException = true;
            });
            services.AddHttpClient();
            services.AddApplicationInsightsTelemetryWorkerService();
            services.ConfigureFunctionsApplicationInsights();
            IConfiguration configuration = hostContext.Configuration;
            services.AddSingleton(configuration);
            var startup = new Startup(configuration);
            startup.ConfigureServices(services);
            services.AddLogging();
        });

}

[Function("Test")] public async Task Test([QueueTrigger("test", Connection = "TestConnectionString")] Custom custom) { try { await CreatePdf(custom); } catch (Exception ex) { await OnFailure(custom, ex); } }

[abkdas123wk]

[abkdas123wk]

Additional Details from Application Insights: Microsoft.Azure.WebJobs.Host.FunctionTimeoutException: at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryHandleTimeoutAsync d35.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:663) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ InvokeWithTimeoutAsync d33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:571) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ ExecuteWithWatchersAsync d32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ ExecuteWithLoggingAsync d26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ ExecuteWithLoggingAsync d26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryExecuteAsync d18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)

Exception Properties Show less Event time 7/4/2024, 4:11:36.1631266 PM (Local time)
Message Timeout value of 00:10:00 was exceeded by function: Functions.Test
Exception type Microsoft.Azure.WebJobs.Host.FunctionTimeoutException
Failed method Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryHandleTimeoutAsync d35.MoveNext
Severity level Error
Telemetry type exception
Operation name Test
Device type PC
Client IP address 0.0.0.0 Role instance pod name SDK version azurefunctions: 4.34.2.2
Sample rate 1
Problem Id Microsoft.Azure.WebJobs.Host.FunctionTimeoutException at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryHandleTimeoutAsync d35.MoveNext
Assembly Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35

[bhagyshricompany]

Thanks for reporting .PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" pls update this as updated one. 1.3.2

[abkdas123wk]

After updating "Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" from Version="1.2.1" to 1.3.2, still the issue persists. We are still getting timeout exception.

[bhagyshricompany]

please use all updated sdk.

[yaseen22]

We have the same issue with Http-trigger Azure function, after migrating from .NET6 to .NET8 Isolated mode, the azure function starts having timeout exception.

And we're using the latest updated packages

[abkdas123wk]

We have updated all packages, still we are getting the timeout exception.

[bhagyshricompany]

will check please share the updated list of pkg with version and memory you used.Thanks

[abkdas123wk]

Below are the list of updated packages: PackageReference Include="Azure.Identity" Version="1.11.2" PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" PackageReference Include="Microsoft.Azure.Core.NewtonsoftJson" Version="2.0.0" PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.3" PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Blobs" Version="6.4.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.2.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.3.2" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.4.0" PackageReference Include="Microsoft.Extensions.Http.Polly" Version="8.0.4" PackageReference Include="Microsoft.Data.SqlClient" Version="5.2.0" PackageReference Include="Microsoft.Extensions.Logging" Version="8.0.0" PackageReference Include="Microsoft.CodeAnalysis.Common" Version="4.9.2" PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.9.2" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" PackageReference Include="NuGet.Packaging" Version="6.9.1" PackageReference Include="RazorEngineCore" Version="2024.4.1" PackageReference Include="System.Security.Cryptography.Xml" Version="8.0.0" PackageReference Include="Microsoft.EntityFrameworkCore" Version="8.0.4" PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.4"

[BGbakker]

Hi, We are experiencing the same issue. After the update of our functions from .net6 in-process to .Net8 isolated, the functions starter throwing timeouts. We are running 4 functions in a P3V3 app service plan. 2 are time triggered, 2 are event hub triggered. All are affected. The timeout seems to occur when the message ingress is higher than normal.

[b-enigma-con]

We have updated all packages, still we are getting the timeout exception.

How does your memory consumption look like?

[yaseen22]

For the Azure functions we didn't check our memory consumption, so I can't tell.

But for our applications, yes after we migrated to .NET 8 we had huge memory consumption that caused us issues. Check this thread, it's helpful. https://github.com/dotnet/runtime/issues/95922

What worked for us was moving our Linux distribution image of our containers to be Alpine instead of Debian.

[anupamk1506]

@yaseen22 , Can you please help me with the Alpine base image that you have used for isolated function with .Net 8 ?

[yaseen22]

@yaseen22 , Can you please help me with the Alpine base image that you have used for isolated function with .Net 8 ?

For Azure functions I didn't use the Alpine image, I'm not sure if this is even possible, For Azure function I uses the .NET environment.

The Alpine base image is used for my application at docker image

[abkdas123wk]

Hi @yaseen22, can you share the docker file you have used for deployment of azure function? We have used the base image 'FROM mcr.microsoft.com/azure-functions/dotnet-isolated:4-dotnet-isolated8.0' for our azure function.

[MathieuMarchant]

Also running into Timeout exceptions here. Found that this is actually caused by enabling the user code exception, as it is documented in the guide here at the bottom of the logging section:

https://learn.microsoft.com/en-us/azure/azure-functions/dotnet-isolated-process-guide?tabs=windows#logging

.ConfigureFunctionsWorkerDefaults(_ => {}, options =>
    {
        // Makes that exceptions are not wrapped in RpcExceptions
        options.EnableUserCodeException = true;
    })

When commenting out this code, everything works as expected.

But the question is then how do we make sure the exceptions are not getting wrapped?

[u4lklusek]

Is there any update around this topic? I have same issue

btw. I do not have the following code piece in my code

.ConfigureFunctionsWorkerDefaults(_ => {}, options => { // Makes that exceptions are not wrapped in RpcExceptions options.EnableUserCodeException = true; })

[kadirercetin]

Is there any update on that issue? We are having the same problem..