aws-samples / experimental-programmatic-access-ccft

Experimental Programmatic Access to the AWS Customer Carbon Footprint Tool data
MIT No Attribution
28 stars 8 forks source link

ERROR - Failed creating database #5

Closed fuellbie closed 5 months ago

fuellbie commented 5 months ago

Hello,

the step function runs successfully. I also don't see any errors in the Lambda execution (since error handling is missing I guess?). However, no Athena Database has been created. In Athena, I see the error under "Recent queries":

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Insufficient Lake Formation permission(s): Required Create Database on Catalog (Service: AmazonDataCatalog; Status Code: 400; Error Code: AccessDeniedException; Request ID: 14419262-b63c-4aeb-8b8b-5df567d98f43; Proxy: null))
This query ran against the "carbon_emissions" database, unless qualified by the query. Please post the error message on our [forum ](https://forums.aws.amazon.com/forum.jspa?forumID=242&start=0)
or contact [customer support ](https://eu-central-1.console.aws.amazon.com/support/home?#/case/create?issueType=technical&serviceCode=amazon-athena&categoryCode=query-related-issue)
with Query Id: e5898119-fef5-4612-bcde-59f9409c5cc3

Not sure how to fix this. Can anyone help here?

ktj-ph commented 5 months ago

Hi Michael, thanks for opening an issue. I cannot replicate this on my side - the set up for the "create-alter-athena-view" Lambda function should grant the necessary permissions for it to interact with Athena and the AWS Glue Data Catalog (see CreateAlterAthenaViewFunction starting l. 270 in template.yaml)

fuellbie commented 5 months ago

Hello, thank you for the quick response! Yes, I can confirm that the error appears for the CREATE DATABASE IF NOT EXISTS carbon_emissions Screenshot from 2024-05-23 14-21-45 I also can confirm that the permissions are set: Screenshot from 2024-05-23 14-24-10

Could it be that the boto3 call start_query_execution takes some default values which may not work in our environment? Since the error mentions lake formation, could it be that something the API call tries to interact with is connected with lake formation? I am not very familiar with the whole Athena/Glue/Lake Formation environment.

Also, what catalog is used by default in this boto3 call?

ktj-ph commented 5 months ago

Error was due to specific lakeformation setup of account. Solved by adding the IAM role used by the "create-alter-athena-view" lambda function to the Database creators in the "Administrative roles and tasks" in Lake Formation. The Athena table was successfully created then.