Performance issue: Slow retrieval times from Parameter Store in .NET 8 Lambda functions

Describe the bug

Hello,

We have observed that Lambda functions using the .NET 8 runtime experience delays when fetching parameters from AWS Systems Manager Parameter Store. The delay ranges from 4 to 15 seconds, depending on the allocated memory size. This behavior is inconsistent and seems to be more pronounced during cold starts.

Regression Issue

[ ] Select this option if this issue appears to be a regression.

Expected Behavior

We expect a consistent and faster response when fetching parameters from the Parameter Store, regardless of the memory configuration or cold/warm start scenarios.

Current Behavior

Below are the observed results during cold and warm starts for both multiple Parameter Stores and a single Parameter Store:

Cold Start (Full Path Parameter Store):

128MB RAM: 18.637 seconds
512MB RAM: 4.30 seconds
1GB RAM: 1.69 seconds
2GB RAM: 0.81 seconds

Warm Start (Full Path Parameter Store):

128MB RAM: 0.439 seconds
512MB RAM: 0.151 seconds
1GB RAM: 0.127 seconds
2GB RAM: 0.113 seconds

Cold Start (Specific Path Parameter Store):

128MB RAM: 18.338 seconds
512MB RAM: 3.85 seconds
1GB RAM: 1.66 seconds
2GB RAM: 0.94 seconds

Warm Start (Specific Path Parameter Store):

128MB RAM: 0.378 seconds
512MB RAM: 0.036 seconds
1GB RAM: 0.079 seconds
2GB RAM: 0.051 seconds

Performance

Reproduction Steps

To reproduce the issue with the slow retrieval times from AWS Systems Manager Parameter Store, follow these steps:
Set up an AWS Lambda function with the .NET 8 runtime.
Use the following code for the Lambda function:

using Amazon.Lambda.Core;
using Microsoft.Extensions.Configuration;
using System.Diagnostics;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace GetSSM.LambdaFunction
{
    public class Function
    {
        public string FunctionHandler(object input, ILambdaContext context)
        {
            var stopWatch = new Stopwatch();
            stopWatch.Start();

            var configuration = GetConfiguration();

            stopWatch.Stop();
            var elapsedTime = stopWatch.ElapsedMilliseconds;

            context.Logger.LogInformation($"Get Parameter Store: {elapsedTime} milliseconds");
            context.Logger.LogInformation($"Get Parameter Store: {elapsedTime / 1000.0} seconds");
            return "Hello World";
        }

        public static IConfiguration GetConfiguration()
        {
            var cb = new ConfigurationBuilder().AddSystemsManager(c =>
                {
                    c.Path = "/my/full/path/";
                });

            return cb.Build();
        }
    }
}

Deploy the function and configure an appropriate IAM role with permission to access Parameter Store.
Use AWS Systems Manager Parameter Store to create parameters under both:
- A specific path (e.g., /my/full/path/parameter).
- A full path structure with multiple parameters.
Test the Lambda function with different memory configurations (e.g., 128MB, 512MB, 1GB, 2GB) and log the results for both:
- Cold starts: Ensure the function is invoked after an idle period or by deploying a new version.
- Warm starts: Invoke the function repeatedly within a short time span.
- Collect the execution times for each scenario and compare the results.

Possible Solution

No response

Additional Information/Context

We’ve also attached a graph visualizing the above results to provide more clarity. This issue seems closely tied to the allocated memory size and the runtime's handling of cold starts.

Key Questions:

Is this behavior expected due to some inherent limitation or design of the library?
Are there any recommendations to mitigate these cold start delays when fetching parameters?
Considering that the function only performs a single GetParameter operation, is it appropriate to scale the memory so significantly to improve performance? Or is there an alternative approach to address this latency without increasing the allocated memory?

Thank you for your assistance! Let us know if further details or tests are required.

AWS .NET SDK and/or Package version used

Amazon.Extensions.Configuration.SystemsManager 6.2.2 Amazon.Lambda.Core 2.2.0 Amazon.Lambda.Serialization.SystemTextJson v

Targeted .NET Platform

.NET 8

Operating System and version

.NET8 Lambda Runtime

aws / aws-dotnet-extensions-configuration