Temporary 'AccessDenied: Missing credentials in config' error

aws-samples / aws-lambda-fanout

A sample AWS Lambda function that accepts messages from an Amazon Kinesis Stream and transfers the messages to another data transport.

Apache License 2.0

288 stars 60 forks source link

Temporary 'AccessDenied: Missing credentials in config' error #12

Closed M1rna closed 7 years ago

M1rna commented 7 years ago

Hi,

we're using the fanout function to push Kinesis Streams' data from one AWS account to another. Looking at the Streams' data throughput in both accounts, it seems to be working correctly. However, when we enabled a Cloud Watch alarm to capture the function's invocation errors, here's what we get periodically: fanout_error fanout_error2

This happens several times a day and seems as a temporary glitch, but we haven't yet had the opportunity to validate if there is any data loss occurring in the process.

Any information you could provide would be appreciated.

Thanks,

julienlepine commented 7 years ago

Do you have any information o how often that error occurs? We regularly refresh the credentials to connect to the outputs, and that may be linked.

Could you provide, for this function:

when the first entry in that log stream appeared
when the error occurred (8:41)
if the error occurred later (and if yes when)
if there is another log stream where this happened, the same information

Thanks, Julien

M1rna commented 7 years ago

Hi Julien,

yes, this seems to be connected to our issue. Further details:

The time of the first entry to the log group in question is 7:41. The error didn't occur later, but we deactivated the triggers for the function at 4 pm, so we didn't monitor it after that time. From 8:41:33 to 8:41:44, the function failed and retried for 5 times altogether, at 8:41:44 succeeded to process that RequestId.

Here are the occurrences of the error for 2017/02/03: fanout_cw

Here's the info on another log stream from that day where this happened 2 times:

time of first entry in the log group: 00:12:17
time of first error: 01:12:06 (failed and retried 5 times, succeeded at 01:12:15)
time of second error: 02:12:06 (failed and retried 3 times, succeeded at 02:12:12)

Thanks,

Mirna

julienlepine commented 7 years ago

Hi Myrna, in the lib/services.js file, could you try a couple changes?

line 43, change the value of the stsSessionDuration variable to 600 (from 3600), that will force credentials refresh every 10 minutes
line 138, change the value 5 to 30 in (stsSessionDuration - 5) to have it as (stsSessionDuration - 30), that will force a refresh 30 seconds before expiration

Could you make these changes and update the defined function and get back to me?

Julien

M1rna commented 7 years ago

Doing that, I got a validation error in the Lambda log saying "Member must have value greater than or equal to 900", so I updated the stsSessionDuration to 900 and left it running. Will keep monitoring it and post an update on the current situation.

Thanks,

Mirna

M1rna commented 7 years ago

Update: the modified function has been running smoothly for the past 2 days.

Thanks a lot for your help and fast response, Julien!