aws-solutions / aws-data-lake-solution

A deployable reference implementation intended to address pain points around conceptualizing data lake architectures that automatically configures the core AWS services necessary to easily tag, search, share, and govern specific subsets of data across a business or with other external businesses.
https://aws.amazon.com/solutions/implementations/data-lake-solution/
Apache License 2.0
401 stars 160 forks source link

Issue creating package #20

Closed coichedid closed 6 years ago

coichedid commented 6 years ago

Hello guys,

After deploy my datalake with CloudFormation template, I logged in DataLake Console and tried to create a new package. I've supplied required parameters and everything apparently worked fine, but behind the scenes I got two 502 errors. Console couldn't find crawler metadata and tables. As I could see, nothing was created in AWS Glue and even if I browse to package edition screen and try to upload a file or even a manifest, nothing happens and I'm not able to save my package.

I really don't know where I can see any glue. I searched every Cloud Watch log entry and found nothing.

Could you help me?

Regards, Clovis Chedid

hvital commented 6 years ago

Hi Clovis,

Did you upload a dataset to this package? Or just created the package and then access it (from search result page or directly link)?

coichedid commented 6 years ago

Hi Clovis,

Did you upload a dataset to this package? Or just created the package and then access it (from search result page or directly link)?

Hello Heitor,

After some tests, I found that if I upload a file and save package, apparently, it works fine. Also, with an empty package, if I just try to use a manifest file, I got an error. Inspecting again all logs, I found that when I upload a manifest file and save package, I got this error: ValidationException: One or more parameter values were invalid: An AttributeValue may not contain an empty string { code: 502, message: 'Failed to creates a new dataset in the data lake and attaches it to package BkfOSpBd7.' }

When I inspect posted content body, I found that content-type is empty. Take a look: body: '{"name":"bucket_pmo.manifest","type":"manifest","content_type":"","owner":"clovis.chedid"}',

My manifest file is followed: { "dataStore": [ { "includePath": "s3://ons-datalake/previsaoCarga/data/pmo/" } ] }

hvital commented 6 years ago

Thanks for your help. Will include the fix for this in the next release.

shsenior commented 6 years ago

Resolved in v2.1.0 update.