Authentication problem with Lambda function

[EloyTolosa]

Hi, I'm having troubles accessing Athena from a Lambda function. I want to use an AccessKeyID and SecretAccessKey from an IAM User I created, which has the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::webbeds-input-zone/datalake-api/*",
                "arn:aws:s3:::datalake-api/*"
            ]
        },
{
            "Sid": "a1",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::aws-athena-query-results-webbeds-datareader"
            ]
        },
        {
            "Sid": "a2",
            "Effect": "Allow",
            "Action": [
                "iam:CreateAccessKey",
                "iam:GetUser",
                "iam:ListAccessKeys"
            ],
            "Resource": [
                "arn:aws:iam::*:user/${aws:username}"
            ]
        },
        {
            "Sid": "a3",
            "Effect": "Allow",
            "Action": [
                "athena:StartQueryExecution",
                "athena:StopQueryExecution",
                "athena:GetQueryResultsStream",
                "athena:GetQueryResults",
                "athena:GetNamedQuery",
                "athena:DeleteNamedQuery",
                "athena:ListQueryExecutions",
                "athena:ListNamedQueries",
                "athena:GetWorkGroup",
                "athena:CreateNamedQuery",
                "athena:GetQueryExecution",
                "athena:BatchGetNamedQuery",
                "athena:BatchGetQueryExecution"
            ],
            "Resource": [
                "arn:aws:athena:eu-west-1:137693930675:workgroup/webbeds_datareader",
                "arn:aws:athena:eu-west-1:137693930675:workgroup/primary"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketPublicAccessBlock",
                "s3:GetBucketPolicyStatus",
                "s3:Get*",
                "athena:GetNamespace",
                "athena:ListWorkGroups",
                "s3:List*",
                "s3:GetBucketAcl",
                "athena:GetCatalogs",
                "athena:GetNamespaces",
                "s3:GetAccountPublicAccessBlock",
                "s3:ListAllMyBuckets",
                "athena:GetExecutionEngine",
                "athena:GetExecutionEngines",
                "athena:GetTables",
                "athena:GetTable",
                "s3:GetBucketLocation",
                "athena:ListWorkGroups",
                "athena:ListDataCatalogs"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::aws-athena-query-results-webbeds-datareader/*",
                "arn:aws:s3:::aws-athena-query-results-*"
            ]
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:CreateDatabase"
            ],
            "Resource": [
                "arn:aws:glue:eu-west-1:137693930675:catalog",
                "arn:aws:glue:eu-west-1:137693930675:database/default"
            ]
        },
        {
            "Sid": "VisualEditor4",
            "Effect": "Allow",
            "Action": "glue:GetTables",
            "Resource": "arn:aws:glue:eu-west-1:137693930675:database/*"
        },
        {
            "Sid": "VisualEditor5",
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:GetPartition",
                "glue:GetTables",
                "glue:GetPartitions",
                "glue:BatchGetPartition",
                "glue:GetDatabases",
                "glue:GetTable"
            ],
            "Resource": [
                "arn:aws:glue:eu-west-1:137693930675:database/default",
                "arn:aws:glue:eu-west-1:137693930675:database/webjet_data_lake",
                "arn:aws:glue:eu-west-1:137693930675:database/webbeds_callcentre",
                "arn:aws:glue:eu-west-1:137693930675:database/webbeds_callcentre/*",
                "arn:aws:glue:eu-west-1:137693930675:table/webjet_data_lake/*",
                "arn:aws:glue:eu-west-1:137693930675:database/webbeds_searches",
                "arn:aws:glue:eu-west-1:137693930675:table/webbeds_searches/*",
                "arn:aws:glue:eu-west-1:137693930675:table/default",
                "arn:aws:glue:eu-west-1:137693930675:table/default/*",
                "arn:aws:glue:eu-west-1:137693930675:table/webbeds_callcentre/*",
                "arn:aws:glue:eu-west-1:137693930675:catalog",
                "arn:aws:glue:eu-west-1:137693930675:table/default/bookings_stream_last",
                "arn:aws:glue:eu-west-1:137693930675:database/webbeds_masterdb",
                "arn:aws:glue:eu-west-1:137693930675:table/webbeds_masterdb/*",
                "arn:aws:glue:eu-west-1:137693930675:database/webbeds_datacc",
                "arn:aws:glue:eu-west-1:137693930675:table/webbeds_datacc/*"
            ]
        },
        {
            "Sid": "CreateEditDropViews",
            "Effect": "Allow",
            "Action": [
                "glue:CreateTable",
                "glue:UpdateTable"
            ],
            "Resource": [
                "arn:aws:glue:eu-west-1:137693930675:catalog",
                "arn:aws:glue:eu-west-1:137693930675:database/webjet_data_lake",
                "arn:aws:glue:eu-west-1:137693930675:table/webjet_data_lake/*"
            ]
        },
        {
            "Sid": "VisualEditor9",
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::webbeds-input-zone*",
                "arn:aws:s3:::webbeds-reporting-zone*",
                "arn:aws:s3:::webbeds-master-zone*",
                "arn:aws:s3:::webbeds-*-datasource*",
                "arn:aws:s3:::webbeds-jactravel-*",
                "arn:aws:s3:::webbeds-sunhotels-*",
                "arn:aws:s3:::webbeds-five9-logs"
            ]
        },
{
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "glue:GetTable",
            "Resource": [
                "arn:aws:glue:eu-west-1:137693930675:catalog",
                "arn:aws:glue:eu-west-1:137693930675:table/webbeds_analysis/pricescrawler_*",
                "arn:aws:glue:eu-west-1:137693930675:database/webbeds_analysis"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "glue:BatchCreatePartition",
            "Resource": [
                "arn:aws:glue:eu-west-1:137693930675:catalog",
                "arn:aws:glue:eu-west-1:137693930675:database/webbeds_analysis",
                "arn:aws:glue:eu-west-1:137693930675:table/webbeds_analysis/pricescrawler_*"
            ]
        },
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::webbeds-sapcheck-logs/uploads/reports/*"
        }
    ]
}

As you can see, there's plenty of permissions to execute queries to athena. Also, I'm unsetting the environment variable AWS_SDK_LOAD_CONFIG by using

os.Unsetenv("AWS_SDK_LOAD_CONFIG")

as the README says.

But, whenever I make a call to athena, the next error message pops up:

I don't know what the problem could be, and this is driving me nuts. I talked to amazon and they told me that maybe this is something related to this driver.

Thanks a lot in advance.

[henrywoo]

Hello @EloyTolosa Can you try the code in https://github.com/uber/athenadriver/tree/master/examples and see if they work? If they work, can do debug and compare the difference between your code and the example code?

[EloyTolosa]

Hi. I have been looking around the code. I saw the link you provided, and I also looked around https://github.com/uber/athenadriver/blob/master/examples/auth.go which contains several authentication methods. In the auth.go file, I cannot see any method that uses what I need. I need the driver to use the AccessKeyID and SecretAccessKey I provide to the driver, which has all permissions to call executions to Athena. What I can only see is that the AccessKeyID and the SecretAccessKey are dummy and do not have a purpose at all.

Am I wrong? Is the only solution to give the Lambda function all the permissions instead of trying to pass it the credentials?

Thanks

[henrywoo]

Can you use it like this? @EloyTolosa https://github.com/uber/athenadriver/blob/master/examples/auth.go#L31

Replace the dummy ones with your real ones.

[EloyTolosa]

Every time I try that method, it returns this message: UnrecognizedClientException: The security token included in the request is invalid I tried with fresh-new AccessKey,SecretAccessKey pair and AccessKey,SecretAccessKey pair given from Assume Role API Call using AWS SDK in Go.

What am I doing wrong? Why does it say that the token included in the request is invalid?

[EloyTolosa]

Hey there. I found the issue. Whenever you try to assume a role in another account, you need to set the AwsSessionToken for your request to work. I did that making a call to the sts.AssumeRole function and passing the session token to the config struct.

Thanks a lot for your time.

I have a suggestion. Maybe write this exact example as there are more people that I'm sure they want to assume a role in another account and have this exact problem. You could put the example in https://github.com/uber/athenadriver/blob/master/examples/auth.go, or at least explaining it in case someone needs it.

[henrywoo]

Great progress @EloyTolosa Can you please share the example code by creating a PR( you can add description in README.md or add the code in auth.go )?

[EloyTolosa]

Sure, I'll do it!