search

Azure / Azure-Functions

1.1k stars 190 forks source link

After migration to .NET 8(Isolated) from .NET 6(In-Proc), Azure Isolated function (queue triggered) throwing timeout issue. #2510

Open abkdas123wk opened 2 weeks ago

abkdas123wk commented 2 weeks ago

The Azure function (queue triggered) is hosted in AKS cluster (Linux Debian). We are migrating from .NET 6 to .NET 8, in this process we migrated this in-proc azure function to isolated function. After migration we started receiving timeout exception whenever we submit 20+ messages to the queue quickly. In .NET 6 (In-Proc), we never faced this issue. This queue triggered isolated function makes some http calls, DB calls, does some processing and generates pdf. These are the settings and packages in .csproj file: PropertyGroup TargetFramework net8.0 AzureFunctionsVersion v4 LangVersion 12.0 OutputType Exe PropertyGroup ItemGroup PackageReference Include="Azure.Identity" Version="1.11.2" PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" PackageReference Include="Microsoft.Azure.Core.NewtonsoftJson" Version="2.0.0" PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.3-preview1" PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Blobs" Version="6.4.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.4.0" PackageReference Include="Microsoft.Extensions.Http.Polly" Version="8.0.4" PackageReference Include="Microsoft.Data.SqlClient" Version="5.2.0" PackageReference Include="Microsoft.Extensions.Logging" Version="8.0.0" PackageReference Include="Microsoft.CodeAnalysis.Common" Version="4.9.2" PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.9.2" PackageReference Include="NuGet.Packaging" Version="6.9.1" PackageReference Include="RazorEngineCore" Version="2024.4.1" PackageReference Include="System.Security.Cryptography.Xml" Version="8.0.0" PackageReference Include="Microsoft.EntityFrameworkCore" Version="8.0.4" PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.4" ItemGroup

Program.cs public class Program { public IConfiguration Configuration { get; private set; } static async Task Main(string[] args) { await CreateHostBuilder(args).Build().RunAsync(); }

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureFunctionsWorkerDefaults(builder =>
        {
            WorkerConfigurationExtensions.UseNewtonsoftJson(builder);
        }, options =>
        {
            options.EnableUserCodeException = true;
        })
        .ConfigureAppConfiguration((hostingContext, config) =>
        {
            var envName = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production";
            var basePath = Directory.GetCurrentDirectory();
            config.SetBasePath(basePath);
            config.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                                    .AddJsonFile($"appsettings.{envName}.json", true, true)
                                    .AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("AxcessEnv")}.json", true, true)
                                    .AddEnvironmentVariables()
                                    .TryAddAzureKeyVault();

            config.Build();
        })
        .ConfigureServices((hostContext, services) =>
        {
            services.Configure<WorkerOptions>(opt =>
            {
                opt.EnableUserCodeException = true;
            });
            services.AddHttpClient();
            services.AddApplicationInsightsTelemetryWorkerService();
            services.ConfigureFunctionsApplicationInsights();
            IConfiguration configuration = hostContext.Configuration;
            services.AddSingleton(configuration);
            var startup = new Startup(configuration);
            startup.ConfigureServices(services);
            services.AddLogging();
        });

}

[Function("Test")] public async Task Test([QueueTrigger("test", Connection = "TestConnectionString")] Custom custom) { try { await CreatePdf(custom); } catch (Exception ex) { await OnFailure(custom, ex); } }

abkdas123wk commented 2 weeks ago

Untitled

abkdas123wk commented 2 weeks ago

Additional Details from Application Insights: Microsoft.Azure.WebJobs.Host.FunctionTimeoutException: at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryHandleTimeoutAsync d35.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:663) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ InvokeWithTimeoutAsync d33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:571) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ ExecuteWithWatchersAsync d32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ ExecuteWithLoggingAsync d26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ ExecuteWithLoggingAsync d26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryExecuteAsync d18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)

Exception Properties Show less Event time 7/4/2024, 4:11:36.1631266 PM (Local time)
Message Timeout value of 00:10:00 was exceeded by function: Functions.Test
Exception type Microsoft.Azure.WebJobs.Host.FunctionTimeoutException
Failed method Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryHandleTimeoutAsync d35.MoveNext
Severity level Error
Telemetry type exception
Operation name Test
Device type PC
Client IP address 0.0.0.0 Role instance pod name SDK version azurefunctions: 4.34.2.2
Sample rate 1
Problem Id Microsoft.Azure.WebJobs.Host.FunctionTimeoutException at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryHandleTimeoutAsync d35.MoveNext
Assembly Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35

bhagyshricompany commented 2 weeks ago

Thanks for reporting .PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" pls update this as updated one. 1.3.2

abkdas123wk commented 1 week ago

After updating "Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" from Version="1.2.1" to 1.3.2, still the issue persists. We are still getting timeout exception.

bhagyshricompany commented 1 week ago

please use all updated sdk.

yaseen22 commented 1 week ago

We have the same issue with Http-trigger Azure function, after migrating from .NET6 to .NET8 Isolated mode, the azure function starts having timeout exception.

And we're using the latest updated packages

image