Glue: Creation of partition indexes fails if the table and account permissions are managed with Lakeformation

ksco92 commented 2 years ago

Describe the bug

After this merge was made:

https://github.com/aws/aws-cdk/pull/17998

We got the ability to add partition indexes, however, if I have a data lake in my account and manage the permissions via Lake Formation, I get the following error on deployment in cloud trail:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "XXXX",
        "arn": "XXXX",
        "accountId": "XXXXX",
        "accessKeyId": "XXXX",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "XXXX",
                "arn": "XXXXX",
                "accountId": "XXXXXX",
                "userName": "XXXXXX"
            },
            "attributes": {
                "creationDate": "2022-08-02T02:17:14Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2022-08-02T02:19:01Z",
    "eventSource": "glue.amazonaws.com",
    "eventName": "CreatePartitionIndex",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "XXXXX",
    "userAgent": "aws-sdk-nodejs/2.1186.0 linux/v14.19.3 exec-env/AWS_Lambda_nodejs14.x promise",
    "errorCode": "AccessDenied",
    "errorMessage": "An unknown error occurred",
    "requestParameters": {
        "databaseName": "dbname",
        "tableName": "tablename",
        "partitionIndex": {
            "keys": [
                "my_col"
            ],
            "indexName": "indexname"
        }
    },
    "responseElements": null,
    "requestID": "XXXX",
    "eventID": "XXXX",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "XXXX",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "clientProvidedHostHeader": "glue.us-east-1.amazonaws.com"
    }
}

And I can see this in the console in the deleted stack:

Received response status [FAILED] from custom resource. Message returned: Insufficient Lake Formation permission(s): Required Alter on tablename (RequestId: XXXX)

From what i can tell the custom resource created by CDK for this operation is deleted upon fail, so I can't see the policy it has. Also not sure how to grant permissions in advance if this doesn't exist.

Expected Behavior

A role/policy can be provided to the table construct so that we can provide LF permissions in advance.

Current Behavior

The custom resource is created with a policy we have no control over, therefore it is not possible to add index partitions to a data lake managed by LF.

Reproduction Steps

It's tricky to add these steps as a whole data lake would have to be created for this.

Possible Solution

A role/policy can be provided to the table construct so that we can provide LF permissions in advance.

Additional Information/Context

No response

CDK CLI Version

1.163.2 (build 1a28d63)

Framework Version

No response

Node.js Version

N/A

OS

N/A

Language

Typescript

Language Version

No response

Other information

No response

kaizencc commented 2 years ago

You're asking for the custom resource role to be exposed so that you can add whatever permisisons necessary to it, right? That shouldn't be too hard in itself. I confess that I never thought of lakeformation when I wrote the custom resource.

ksco92 commented 2 years ago

Yes! If after creating the table we could access the role through an attribute of the table, we could grant the ALTER permission through the Lake Formation constructs.

ksco92 commented 1 year ago

Are there any updates on this? LF is now pretty standard in terms of data lake permission management.

aws / aws-cdk