Async methods fail

[Raphael-Saldanha]

After upgrading to the latest version, the EncryptAsync thread remais locked

using (var client = new AmazonKeyManagementServiceClient())
using (var plainText = new MemoryStream(Encoding.UTF8.GetBytes("Amazon Web Services")))
{
    var request = new EncryptRequest
    {
        KeyId = "Aws-Kms-Login-Key",
        Plaintext = plainText
    };

    var result = client.EncryptAsync(request).Result.CiphertextBlob.ToArray();
}

To overcome the issue, it is required to run the code within a different task

var result = Task.Run(() => _ = client.EncryptAsync(request).Result).Result.CiphertextBlob.ToArray();

[Raphael-Saldanha]

The same problem can be found on AmazonS3Client's PutBucketAsync method.

[Raphael-Saldanha]

On a Asp.NET MVC Application, everything works as expected having the following package versions

• AWSSDK.Core: 3.3.107.5
• AWSSDK.KeyManagementService: 3.3.106.1

if the AWSSDK.Core updated from version 3.3.107.5 to 3.3.107.6 the AmazonKeyManagementServiceClient's EncryptAsync method thread will be blocked, I mean, it will never return any result.

The issue is also present by updating the AWSSDK.KeyManagementService to the 3.3.106.2, that as a requirement will also update the AWSSDK.Core to 3.3.107.6.

The issue is only present in Asp.NET MVC application. It is not possible to reproduce the issue on a Console Application running on both .NET Framework and .NET Core, using the packages on the versions mentioned above or even on their latest versions.

public class HomeController : Controller
{
    public ActionResult Index()
    {
        using (var client = new AmazonKeyManagementServiceClient("AKIA...", "KEY...", RegionEndpoint.USEast1))
        using (var plainText = new MemoryStream(Encoding.UTF8.GetBytes("Amazon Web Services")))
        {
            var request = new EncryptRequest
            {
                KeyId = "arn:aws:kms:us-east-1:XXXXXX:key/6756AB19-XXXX-XXXX-XXXX-4D00324E28BF",
                Plaintext = plainText
    ``        };

            var result = client.EncryptAsync(request).Result.CiphertextBlob.ToArray();
        }

        return View();
    }
}

[ukmoredec]

I'm experiencing the same thing in a Lambda function project w/ .net core 3.1, with a call to AmazonElastiCacheClient.DescribeReplicationGroupsAsync. Wrapping the call in a new Task like in the OP "fixes" the problem.

This is with Core 3.3.107.10 and ElastiCache 3.3.111.5--and a lot of other packages and mess working with older versions, I also can't reproduce it in a fresh Lambda function with the same two package versions above. This doesn't include a bunch of other services/configuration that the main project has, though. I'll try harder to come up with the smallest project that reproduces the problem, just wanted to +1 this for now.

[rajeshsv]

We have the same issue after upgrading to the latest version of nuget.

After spending quite a lot of time investigating we found the issue in AWSSDK.Core latest nuget where the async call does not return to the calling thread.

I have done a very basic sample application where using the AWSSDK.Core v 3.3.107.3 returns the call back but when we use the latest version the call does not come back

    protected void Page_Load(object sender, EventArgs e)
    {
        GetValueFromSSM();
    }

    private void GetValueFromSSM()
    {
        try
        {
            var accessKey = "";
            var secretKey = "";
            var parameterStoreKey = "";
            var objSsm = new AmazonSimpleSystemsManagementClient(accessKey, secretKey, Amazon.RegionEndpoint.EUWest1);
            var ssmRequest = new GetParameterRequest() { Name = parameterStoreKey, WithDecryption = false };
            // The below code works when  AWSSDK.Core is 3.3.107.3 
            var ssmValue = objSsm.GetParameterAsync(ssmRequest).GetAwaiter().GetResult();
        }
        catch (Exception ex)
        {
            throw ex;
        }
    }

The very simple example as above can be used to reproduce the error.

UPDATE 24 June 2020: Just had a update from the AWS Support team that they have release a new version of AWSDK.Core that has the issue fixed.

[ukmoredec]

I cobbled together a ~minimal reproduction from our use case AWSLambda2.zip

ConfigureAwait(false) on the await ...Async call gets around the problem as well.

Interestingly, running this with the Mock Lambda Test Tool with --no-ui also gets around the problem!

[boblodgett]

This sounds like another issue that was happening in ASP.NET web applications where the async calls were hanging. I released a fix for this today as of versions: AWSSDK.Core 3.3.107.16 AWSSDK.KeyManagementService 3.3.106.12

Can you give these versions a try and let us know if it corrected this issue? This should fix the async calls from hanging.

[boblodgett]

Closing this issue as the fix released should correct the problem and I have not heard back. Please reopen if the issue persists after upgrading to the latest versions.

[ukmoredec]

Hi Bo, afraid I need to reopen. I've updated my minimal repro to use the latest versions, unless there are some hidden dependencies I'm missing, and the deadlock remains unless I ConfigureAwait(false) on every sdk call (which I guess I should do, but didn't need to before), or run with --no-ui. Do you mind taking a look at AWSLambda2.zip and let me know if I'm doing something wrong? Thanks!

[danielmarbach]

Maybe this helps https://github.com/aws/aws-sdk-net/pull/1651

[SrutiG]

Thank you for the helpful PR, @danielmarbach. Though we have merged it in and it fixes other probable issues with our asynchronous logic, it doesn't seem to fix the issue for @ukmoredec so I am reopening this.

[JeffAshton]

@SrutiG At D2L we use a Roslyn analyzer to enforce this. It might be worth incorporating something similar into this project:

https://github.com/Brightspace/D2L.CodeStyle/blob/master/src/D2L.CodeStyle.Analyzers/Language/AwaitedTasksAnalyzer.cs

[danielmarbach]

It is built into fxcop. I would recommend that even if you disable all the other rules https://docs.microsoft.com/en-us/visualstudio/code-quality/ca2007?view=vs-2019

[JeffAshton]

Looks like it might just be disabled

https://github.com/aws/aws-sdk-net/blob/b122d89177c2e9f220c1f245086c45ab9f5f8385/sdk/AWSDotNetSDK.ruleset#L73

[normj]

I pushed out version 0.10.1 of the .NET Lambda Mock Test Tool today that fixes the issue of it hanging if the Lambda code calls some blocking async code or a missing ConfigureAwait(false);

If you are using Visual Studio the next time you restart visual studio and debug it should download the new version. To confirm you have the newest version you can run the command in the command window.

dotnet tool list -g

You should see 0.10.1 for amazon.lambda.testtool-3.1.

If for some reason you don't you can manually update running the following command:

dotnet tool update -g amazon.lambda.testtool-3.1

[ukmoredec]

@normj Thanks, this solves the problem for me!

[NGL321]

Closing for resolution

[JeffAshton]

But you still haven't solved the systemic problem, and another case like this can easily sneak back in. 😞

[danielmarbach]

They just need to finally merge https://github.com/aws/aws-sdk-net/pull/1659