search

aws-solutions / aws-data-lake-solution

A deployable reference implementation intended to address pain points around conceptualizing data lake architectures that automatically configures the core AWS services necessary to easily tag, search, share, and govern specific subsets of data across a business or with other external businesses.
https://aws.amazon.com/solutions/implementations/data-lake-solution/
Apache License 2.0
389 stars 160 forks source link

Datalake 2.1 fails on non-federated stack. (data-lake-deploy) #22

Closed ironsheiky closed 4 years ago

ironsheiky commented 5 years ago

When deploying the latest version of the Data Lake Solution, not using ADFS, it fails to deploy on any account. Seems that the method for Cognito deployment has changed, assuming it has something to do with that. (ref: https://aws.amazon.com/answers/big-data/data-lake-solution/)

'ResourceNotFoundException: Policy with name [data-lake-es-logs-us-west-2-123456789012] does not exist.\n at Request.extractError

hvital commented 5 years ago

Hi @ironsheiky. Trying to reproduce it here.

Do you have access to create a Support Case in your AWS Console? It would help to exchange some information in order to troubleshoot this faster.

carlsoi commented 5 years ago

This build fails for me as well on a brand-new empty account.

kurtbischoffjr commented 5 years ago

Can confirm. We are also getting this same error. Our AWS rep tried to install into his environment which has full admin privileges ALSO faced same error. Please get this resolved ASAP, its preventing us from trying out this architecture, and is easily repeatable.

hvital commented 5 years ago

Hi @kurtbischoffjr.

Is it still trowing an error? I've launched a couple of times (triple checked one more time now) and seems that I need your help to troubleshoot in your environment and/or using your input parameters.

Do you have access to open support tickets inside your AWS Console? If yes, please open and ask to cc me (Heitor Vital - heitorc).

kurtbischoffjr commented 5 years ago

Please work with Billy Harris. He is our AWS rep for State Farm. He got the same error I got in his environment, and he can perhaps work more directly with you to resolve our issue.

https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon Virus-free. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Tue, Dec 25, 2018 at 4:59 AM Heitor Vital notifications@github.com wrote:

Hi @kurtbischoffjr https://github.com/kurtbischoffjr.

Is it still trowing an error? I've launched a couple of times (triple checked one more time now) and seems that I need your help to troubleshoot in your environment and/or using your input parameters.

Do you have access to open support tickets inside your AWS Console? If yes, please open and ask to cc me (Heitor Vital - heitorc).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/awslabs/aws-data-lake-solution/issues/22#issuecomment-449834521, or mute the thread https://github.com/notifications/unsubscribe-auth/Ar5PaYYm6ZzGUjNdTZL9MF1c6nuyqzCMks5u8fcSgaJpZM4ZXmRm .

mjh7 commented 5 years ago

I am trying to deploy AWS Data Lake solution template on a free-tier account to get familiar with the architecture prior to an official POC for my company.

I have tried several times and it always fails on the same step and rolls back. I don't have the ability to do an AWS support ticket yet so any ideas would be appreciated as I can't really do anything until I figure this out... thanks!

The CloudFormation event log shows a bunch of stuff successful and then this is the start of the errors:

[Type] Custom::LoadLambda

[Logical ID] DakaLakeKibanaCognito

[Status Reason] Failed to create resource. https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/lambda/data-lake-helper;stream=2019/01/10/[$LATEST]d5b673d10d9d4b5fa3ca372aa166f118

hvital commented 5 years ago

Hi @mjh7

Thanks for the feedback!

The similar cases that I've helped to troubleshoot so far was failing because the defined value for Amazon Cognito Domain was invalid or unavailable. As it is used to configure the Cognito's domain prefix, it must be unique across the chosen AWS region and can only contain lowercase letters, numbers, and hyphens.

Try to set something like dl-region-code-mjh7-short-random-sufix. If you still get the error on CloudFormation event log, please follow the cloudwatch logs link printed (https://console.aws.amazon.com/cloudwatch/home ...). It must details the root cause of the error.

wderezin commented 5 years ago

To address a couple issues in this thread. If you deploy (or attempt to deploy) the stack multiple time the cognito user pool does not delete. Probably why @hvital adding a suffix works.

Also, if your account enforces S3 with KMS encryption this is not supported. The work around is to add KMS permissions to the data-lake-helper-role. But wait, there is more since CloudFormations does not support S3 KMS either. So you need to serve the datalakeweb S3 content with an alternate solution such as a Lambda proxy behind an API-Gateway or Application Load Balancer. Then update the DynamoDB setting file so the email invitation are sent with the correct email address.

@mjh7 note that deploying this into a free tier account will still generate a bill in excess of $200 a month due to the ElasticSearch deployment. If you want to reduce that cost drastically modify the data-lake-storage.yaml file to use a smaller instance for the ES service.

mjh7 commented 5 years ago

Thanks to @hvital and @wderezin!

I cleared out what I could and made a more unique Cognito Domain name and the deployment succeeded.

That was before I read about the elastic search charges... I will try to figure out if there is a way to scale down the existing ES service instance, but if you already know if this will cause issues or can't be done after deployment I'd appreciate a heads-up and I will clear out and start over again with a modified template.

knihit commented 4 years ago

Hi @mjh7 if the above issue is resolved, can we close this issue. If you have any more questions, please feel free to reach out

mjh7 commented 4 years ago

Yes that resolved it - thx again.

On Thu, Dec 19, 2019, 1:41 PM knihit notifications@github.com wrote:

Hi @mjh7 https://github.com/mjh7 if the above issue is resolved, can we close this issue. If you have any more questions, please feel free to reach out

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/awslabs/aws-data-lake-solution/issues/22?email_source=notifications&email_token=AIEHKCPTJYCKBE76QGRKY4TQZPTCLA5CNFSM4GK6MRTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHLDZAA#issuecomment-567688320, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIEHKCLMWRAUFZZB7FKWYYDQZPTCLANCNFSM4GK6MRTA .

kurtbischoffjr commented 4 years ago

I abandoned this solution even after getting it working in favor of lake formation. I dont personally find their "lake solution" a viable option.

On Wed, Jan 9, 2019, 8:06 PM mjh7 notifications@github.com wrote:

I am trying to deploy AWS Data Lake solution template on a free-tier account to get familiar with the architecture prior to an official POC for my company.

I have tried several times and it always fails on the same step and rolls back. I don't have the ability to do an AWS support ticket yet so any ideas would be appreciated as I can't really do anything until I figure this out... thanks!

The CloudFormation event log shows a bunch of stuff successful and then this is the start of the errors:

[Type] Custom::LoadLambda

[Logical ID] DakaLakeKibanaCognito

[Status Reason] Failed to create resource. https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/lambda/data-lake-helper;stream=2019/01/10/[$LATEST]d5b673d10d9d4b5fa3ca372aa166f118

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/awslabs/aws-data-lake-solution/issues/22#issuecomment-452931757, or mute the thread https://github.com/notifications/unsubscribe-auth/Ar5PaezkCi6iMjPSU3PBh-gqXXoBZnrUks5vBpIUgaJpZM4ZXmRm .