IAM role permissions issues?

[russAIC]

When I run this:

user@ip-168-0-1-100$python provision_tables.py provisioning_whitelist.hjson
Building Table List for Processing from provisioning_whitelist.hjson
Using compiled configuration config.hjson
Loaded configuration from config.hjson
Resolved DynamoDB Stream ARN: arn:aws:dynamodb:us-west-2:xxx:table/Questionnaire/stream/2017-09-12T22:15:29.068
Resolved Firehose Delivery Stream ARN: arn:aws:firehose:us-west-2:xxx:deliverystream/Questionnaire
Exception while provisioning table Questionnaire
An error occurred (InvalidParameterValueException) when calling the CreateEventSourceMapping operation: Cannot access stream arn:aws:dynamodb:us-west-2:xxx:table/Questionnaire/stream/2017-09-12T22:15:29.068. Please ensure the role can perform the GetRecords, GetShardIterator, DescribeStream, and ListStreams Actions on your stream in IAM.
Proceeding...

The block starting with Resolved Firehose to proceeding... seems to repeat for all tables in my collection.

From what I can tell the cloudWatchRoleArn, firehoseDeliveryRoleArn, and lambdaExecRoleArn have the correct permissions. They look similar to the permissions policies shown in the appendix. Is there an additional configuration step that I've missed? Perhaps something specific to the firehose setup??

[russAIC]

In my LambdaExecRole I've attached two policies the AWSLambdaFullAccess and AmazonKinesisFirehoseFullAccess both of which are AWS managed IAM policies and that should give the actions listed in the appendix:

            "Action": [
                "firehose:CreateDeliveryStream",
                "firehose:DescribeDeliveryStream",
                "firehose:ListDeliveryStreams",
                "firehose:PutRecord",
                "firehose:PutRecordBatch",
                "dynamodb:DescribeStream",
                "dynamodb:DescribeTable",
                "dynamodb:GetRecords",
                "dynamodb:GetShardIterator",
                "dynamodb:ListStreams",
                "dynamodb:ListTables",
                "dynamodb:UpdateTable",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "lambda:CreateFunction",
                "lambda:CreateEventSourceMapping",
                "lambda:ListEventSourceMappings",
                "iam:passrole",
                "s3:Get*",
                "s3:List*"

[IanMeyers]

Was the above policy attached when you tried to setup the configuration first and this error was generated?

[russAIC]

I've gotten a bit further, by removing and trying again.

It seems like this error is what's actually blocking progress more than the above:

2017-09-21T22:05:34.989Z 2c5b2ce2-11ad-4b56-869d-3e74f945a609 { [ResourceNotFoundException: Firehose LambdaStreamsDefaultDeliveryStream not found under account xxx.] message: 'Firehose LambdaStreamsDefaultDeliveryStream not found under account xxx.', code: 'ResourceNotFoundException', time: Thu Sep 21 2017 22:05:34 GMT+0000 (UTC), requestId: 'fe23f3d6-9f18-11e7-b485-97e56b7be019', statusCode: 400, retryable: false, retryDelay: 0.5356467794626951 }

My understanding is that this stream should be created as part of these steps:

cd scripts/dynamodb-continuous-backup/ ls cd src/ python deprovision_tables.py provisioning_whitelist.hjson ./build.sh config.hjson python deploy.py --config-file config.hjson

aws dynamodb create-table --region us-west-2 --attribute-definitions AttributeName=MyHashKey,AttributeType=S --key-schema AttributeName=MyHashKey,KeyType=HASH --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 --table-name MyFourthTest

When i test the lambda function inside aws console this is the output I get:

START RequestId: bbf4e936-9f1c-11e7-822a-fbe568b43e61 Version: $LATEST 2017-09-21T22:32:21.974Z bbf4e936-9f1c-11e7-822a-fbe568b43e61 Processing Complete 2017-09-21T22:32:21.975Z bbf4e936-9f1c-11e7-822a-fbe568b43e61 OK 2017-09-21T22:32:21.975Z bbf4e936-9f1c-11e7-822a-fbe568b43e61 Event contains no Data 2017-09-21T22:32:21.975Z bbf4e936-9f1c-11e7-822a-fbe568b43e61 {"errorMessage":"ERROR"} END RequestId: bbf4e936-9f1c-11e7-822a-fbe568b43e61 REPORT RequestId: bbf4e936-9f1c-11e7-822a-fbe568b43e61 Duration: 0.65 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 55 MB

It seems like there may be an issue with the linkages of the function to the rest of the pipeline, but I can't quite figure out where/how.

[russAIC]

I removed everything triggers, iam role, lambda function, and deleted the S3 bucket. I then re-installed being extremely careful to follow read me exactly. After install three tables out of 20 show up in S3, but i don't think the data is being updated. Only a few of the dynamo db tables have the trigger attached to them. Even though all of them look to be gathered when running the provisioning script:

python provision_tables.py provisioning_whitelist.hjson Building Table List for Processing from provisioning_whitelist.hjson Using compiled configuration config.hjson Loaded configuration from config.hjson Resolved DynamoDB Stream ARN: arn:aws:dynamodb:us-west-2:xxx:table/Questionnaire/stream/2017-09-12T22:15:29.068 Resolved Firehose Delivery Stream ARN: arn:aws:firehose:us-west-2:xxx:deliverystream/Questionnaire Deploying LambdaStreamToFirehose/LambdaStreamToFirehose-1.4.5.zip from s3://awslabs-code-us-west-2 Created New Function LambdaStreamToFirehose:arn:aws:lambda:us-west-2:xxx:function:LambdaStreamToFirehose Resolved DynamoDB Stream ARN: arn:aws:dynamodb:us-west-2:xxx:table/SampleMetadataItems/stream/2017-09-12T22:15:32.296 Resolved Firehose Delivery Stream ARN: arn:aws:firehose:us-west-2:xxx:deliverystream/SampleMetadataItems

When I test the lambda function it gives an error: START RequestId: 18ed700e-a467-11e7-9ad1-dbdee1abba86 Version: $LATEST 2017-09-28T16:07:16.534Z 18ed700e-a467-11e7-9ad1-dbdee1abba86 Processing Complete 2017-09-28T16:07:16.553Z 18ed700e-a467-11e7-9ad1-dbdee1abba86 OK 2017-09-28T16:07:16.553Z 18ed700e-a467-11e7-9ad1-dbdee1abba86 Event contains no Data 2017-09-28T16:07:16.554Z 18ed700e-a467-11e7-9ad1-dbdee1abba86 {"errorMessage":"ERROR"} END RequestId: 18ed700e-a467-11e7-9ad1-dbdee1abba86 REPORT RequestId: 18ed700e-a467-11e7-9ad1-dbdee1abba86 Duration: 28.53 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 39 MB

What I find interesting is that I think the issue is related to the lambda function being v1.4.5. I've tried a manual install of 1.5.1, but not had any luck. I think this is related: https://github.com/awslabs/lambda-streams-to-firehose/pull/54

When I change LAMBDA_STREAMS_TO_FIREHOSE_VERSION to ="1.5.1" this is what happens: An error occurred (AccessDeniedException) when calling the CreateFunction operation: Your access has been denied by S3, please make sure your request credentials have permission to GetObject for awslabs-code-us-west-2/LambdaStreamToFirehose/LambdaStreamToFirehose-1.5.1.zip. S3 Error Code: AccessDenied. S3 Error Message: Access Denied`

[IanMeyers]

Can you please try again? Version 1.5.1 is now uploaded everywhere and I've confirmed permissions.

[IanMeyers]

And please note that this utility will only work in regions where kinesis firehose is supported. See https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/ for more information

[russAIC]

I'm still getting permissions issues when I attempt to read from your bucket:

Resolved DynamoDB Stream ARN: arn:aws:dynamodb:us-west-2:xxxx:table/trial_patient_events_history/stream/2017-09-12T22:16:16.748
Resolved Firehose Delivery Stream ARN: arn:aws:firehose:us-west-2:xxxx:deliverystream/trial_patient_events_history
Deploying LambdaStreamToFirehose/LambdaStreamToFirehose-1.5.1.zip from s3://awslabs-code-us-west-2
Exception while provisioning table trial_patient_events_history
An error occurred (AccessDeniedException) when calling the CreateFunction operation: Your access has been denied by S3, please make sure your request credentials have permission to GetObject for awslabs-code-us-west-2/LambdaStreamToFirehose/LambdaStreamToFirehose-1.5.1.zip. S3 Error Code: AccessDenied. S3 Error Message: Access Denied
Proceeding...

I was able to download the 1.5.1 file from the git repo and I've placed it into a bucket in my account called "from-awslabs-code". I've also changed the dynamo_continuous_backup script to read:

LAMBDA_STREAMS_TO_FIREHOSE = "LambdaStreamToFirehose"
LAMBDA_STREAMS_TO_FIREHOSE_VERSION = "1.5.1"
LAMBDA_STREAMS_TO_FIREHOSE_BUCKET = "from-awslabs-code"
LAMBDA_STREAMS_TO_FIREHOSE_PREFIX = "LambdaStreamToFirehose"

However, attempting to run the script gives errors on connecting to the bucket. The script gives:

Deploying LambdaStreamToFirehose/LambdaStreamToFirehose-1.5.1.zip from s3://from-awslabs-code-us-west-2

Then,

An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: Error occurred while GetObject. S3 Error Code: NoSuchBucket. S3 Error Message: The specified bucket does not exist: InvalidParameterValueException

So as a work around I've attempted to hard code the bucket into the script:

Code={
  'S3Bucket': "from-awslabs-code/LambdaStreamToFirehose/",
  'S3Key' : "LambdaStreamToFirehose-1.5.1.zip"
},

This however is giving a different error:

An error occurred (ValidationException) when calling the CreateFunction operation: 1 validation error detected: Value 'from-awslabs-code/LambdaStreamToFirehose/' at 'code.s3Bucket' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[0-9A-Za-z\.\-_]*(?<!\.)$: ClientError

When I attempted to use the full URL for the S3Bucket it gave the same error.

I'm working in US-West-2 (OR) region, so all items should be supported.

In the process I remove the Lambda Function and the Cloud Watch Event Rule and allow the scripts to re-create. Also when running the deploy.py script I use the --redeploy flag along with the config file.

Thoughts?

[IanMeyers]

I've confirmed that 1.5.1 is everywhere and that it's public readable. I've also bumped the version in ddb continuous backup to 1.5.1. Can you try and resync from github and then retry? If that still doesn't work, please drop me a line on meyersi@amazon.com and maybe we can have a phone call or something.