search

aws-samples / data-engineering-for-aws-immersion-day

Lab Instructions for Data Engineering Immersion Day
MIT No Attribution
177 stars 95 forks source link

Incremental Data Processing with Hudi Failing #27

Closed vayu068 closed 2 years ago

vayu068 commented 2 years ago

S3 holding Dependent jars isn't accesible. Please find below error logs LAUNCH ERROR | Error downloading from S3 for bucket: aws-dataengineering-day.workshop.aws, key: hudijars/calcite-core-1.10.0.jar.Access Denied (Service: Amazon S3; Status Code: 403; Please refer logs for details.

melodyyangaws commented 2 years ago

it should be OK now. please try again and please let me know if you still can't access it.

tomaszdudek7 commented 2 years ago

@melodyyangaws We've tried running this recently with @Ivnnk (i.e. yesterday) and the 12th point in Step 1 is what makes the job fail. There's no access to that bucket.

melodyyangaws commented 2 years ago

Thanks for the details @tomaszdudek7 . that's really helpful for us to locate the root cause. The permission issue is fixed now. please give it a try with the updated CFN template or manually add the s3 bucket permission to your GlueLabRole in your existing lab environment. You can see the detail in the PR request.

IVnnk commented 2 years ago

I can confirm everything works. Thanks!