AmazonLambdaClient.Invoke() throws AmazonLambdaException due to Expired Signature

devangelito commented 6 years ago

We have a lambda function developed in C# (dotnetcore) that invokes another lambda function. Intermittently, we are getting this Amazon.Lambda.AmazonLambdaException from AmazonLambdaClient.Invoke() due to expired signature (internally thrown via Amazon.Runtime.Internal.HttpErrorResponseException).

Expected Behavior

AmazonLambdaClient.Invoke() does not encounter this exception intermittently. Not sure if clock-skew correction is also in place for Lambda functions, but internal retry mechanism would be great.

Current Behavior

Intermittently, invocation of lambda function from a lambda function results to exception. Stack trace below:

Error: Amazon.Lambda.AmazonLambdaException: Signature expired: 20171127T214626Z is now earlier than 20171127T215544Z (20171127T220044Z - 5 min.) ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
at Amazon.Runtime.HttpWebRequestMessage.<GetResponseAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__3`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext()
--- End of inner exception stack trace ---
at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at xxx.xxx.xxx.xxx.xxx.<Method3>d__2.MoveNext() in /.../.../.../.../.../.../.../.../CodeC.cs:line xx
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at xxx.xxx.xxx.xxx.<Method2>d__4.MoveNext() in /.../.../.../.../.../.../.../CodeB.cs:line xx
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at xxx.xxx.xxx.xxx.<Method1>d__7.MoveNext() in /.../.../.../.../.../.../.../CodeA.cs:line xxx

Steps to Reproduce (for bugs)

Happens intermittently, so can't really provide steps to ensure it happens. On the other hand, the below code snippet is used:

using (var client = new AmazonLambdaClient())
{
    var request = new InvokeRequest
    {
        FunctionName = subscriber.LambdaArn,
        Payload = JsonConvert.SerializeObject(@event)
    };

        // This below throws the exception
    var response  = await client.InvokeAsync(request);
}

Possible Solution

Manually implement retry for Lambda invocation whenever this exception occurs.

Your Environment

AWSSDK.Core version used: 3.3.18.12
Service assembly and version used: AWSSDK.Lambda 3.3.8
Operating System and version: Windows 10 (64-bit)
Visual Studio version: 2017
Targeted .NET platform: .NETCoreApp 1.0

sstevenkang commented 6 years ago

.NET SDK already does clockskew correction by default. Could you provide request ids for the requests that results in expired signature eception?

normj commented 6 years ago

Out of curiosity how long does your Lambda function run and what is the timeout set on your Lambda function?

devangelito commented 6 years ago

Hello @sstevenkang, Below are sample request ids:

4e7292cc-d9bc-11e7-bb93-43bcee7d169c (5 Dec 2017 13:00 UTC)
54b9a1d2-d8a3-11e7-8cac-53e627ca912b (4 Dec 2017 3:29 UTC)
4bbeff75-d622-11e7-b93c-0767bce9b1e8 (30 Nov 2017 23:00 UTC)

To answer @normj, we have currently set timeout to 5 minutes.

aws / aws-sdk-net