search

fullstorydev / hauser

Service for moving your Fullstory export files to a data warehouse
MIT License
49 stars 23 forks source link

Failure to upload files from s3 to Amazon RedShift #10

Closed alexfowler closed 5 years ago

alexfowler commented 6 years ago

When running the hauser script, I get the following error:

Failed to load file '/tmp/5724681974841344.csv' to warehouse: pq: User arn:aws:redshift:us-west-1:264207711400:dbuser:us-aws-nk-redshift/test_user is not authorized to assume IAM Role arn:aws:iam::264207711400:role/ec2_s3_fullstory_data_export

I believe has to do with the IAM role credentials in the config.toml, however the example-config.toml file is rather vague on what a correct example should look like and there's no documentation on how to properly configure such an IAM role for the hauser script (that i know of).

So far I'm using this as my main reference on configuring the IAM role: https://docs.aws.amazon.com/redshift/latest/mgmt/generating-iam-credentials-role-permissions.html

Below is the policy that I attached to the above IAM role on the AWS documentation:

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "redshift:GetClusterCredentials",
        "Resource": "*"
    }
}
joelash commented 5 years ago

@alexfowler I know this was almost a year ago, but I found out today that the Credentials config is not used for uploading to S3, instead it uses your current session. So either use ~/.aws/credentials or set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

alexfowler commented 5 years ago

@joelash thank you, I managed to get it working in the end but neglected to comment back here. I believe it was per your suggested solution. Closing the ticket.