data-lake-deploy.template does not exist in "dist" folder after build

aws-solutions / aws-data-lake-solution

A deployable reference implementation intended to address pain points around conceptualizing data lake architectures that automatically configures the core AWS services necessary to easily tag, search, share, and govern specific subsets of data across a business or with other external businesses.

https://aws.amazon.com/solutions/implementations/data-lake-solution/

Apache License 2.0

401 stars 160 forks source link

data-lake-deploy.template does not exist in "dist" folder after build #26

Closed shaoliang-jia-dragon closed 4 years ago

shaoliang-jia-dragon commented 5 years ago

Hi,

After this step "Build the data lake solution for deployment", I didn't find "data-lake-deploy.template" or "data-lake-deploy-federated.template" in the dist folder.

I tried to copy the "data-lake-deploy.template" and "data-lake-deploy-federated.template" to "dist" folder and then upload to s3 bucket. However, I got "TemplateURL must be an Amazon S3 URL" error when running "Deploy the data lake solution" using Cloudformation.

Here are my environment variables: export AWS_REGION=us-east-1 export VERSION_CODE=v2.0.0 export DEPLOY_BUCKET=jslake

timcleaver commented 5 years ago

Not sure if it still matters, but lines 54-57 of deployment/build-s3-dist.sh refer to data-lake-deploy.yaml and data-lake-federated.yaml to be copied into dist/. These files should actually be data-lake-deploy.template and data-lake-federated.template. I have attached a patch to fix this. It also adds set -e, set -u and set -o pipefail bash options which should cause the script to fail if any subcommands fail. There are a number of other smaller changes to handle other failures that set -e highlighted. build-s3-dist.sh.txt

shaoliang-jia-dragon commented 5 years ago

Thanks. Will verify later.

On Thu, 28 Mar 2019 at 12:06, Tim Cleaver notifications@github.com wrote:

Not sure if it still matters, but lines 54-57 of deployment/build-s3-dist.sh refer to data-lake-deploy.yaml and data-lake-federated.yaml to be copied into dist/. These files should actually be data-lake-deploy.template and data-lake-federated.template. I have attached a patch to fix this. It also adds set -e, set -u and set -o pipefail bash options which should cause the script to fail if any subcommands fail. There are a number of other smaller changes to handle other failures that set -e highlighted. build-s3-dist.sh.txt https://github.com/awslabs/aws-data-lake-solution/files/3016129/build-s3-dist.sh.txt

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/awslabs/aws-data-lake-solution/issues/26#issuecomment-477439684, or mute the thread https://github.com/notifications/unsubscribe-auth/AtyhOYQNkPiHYvNH8N2vc2o9OM4F-oI3ks5vbD-vgaJpZM4bbcz5 .

sardbaba commented 5 years ago

@shaoliang-jia-dragon please see https://github.com/awslabs/aws-data-lake-solution/pull/21 I think you can close this issue.

shaoliang-jia-dragon commented 5 years ago

Which issue is it related to?

On Tue, 5 Nov 2019 at 21:16, Mauro Mascia notifications@github.com wrote:

@shaoliang-jia-dragon https://github.com/shaoliang-jia-dragon please see #21 https://github.com/awslabs/aws-data-lake-solution/pull/21 I think you can close this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/awslabs/aws-data-lake-solution/issues/26?email_source=notifications&email_token=ALOKCOKEPZ74BOGKDYIFQADQSFW2NA5CNFSM4G3NZT42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDCY3SY#issuecomment-549817803, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALOKCOMVKBSQRQE6PUMXRVTQSFW2NANCNFSM4G3NZT4Q .

sardbaba commented 5 years ago

Yes sorry, your issue has two parts:

the title of this issue refers to the same issue resolved here https://github.com/awslabs/aws-data-lake-solution/pull/21
"TemplateURL must be an Amazon S3 URL" seems another kind of problem. Did you copy the code with: aws s3 cp ./dist s3://$DEPLOY_BUCKET-$AWS_REGION/data-lake/$VERSION_CODE --recursive --acl bucket-owner-full-control including $VERSION_CODE in the end? Because this "detail" is missing in the Readme.

shaoliang-jia-dragon commented 5 years ago

I didn't raise this issue. 🤐

On Thu, 7 Nov 2019 at 00:17, Mauro Mascia notifications@github.com wrote:

Yes sorry, your issue has two parts:

the title of this issue refers to the same issue resolved here #21 https://github.com/awslabs/aws-data-lake-solution/pull/21

"TemplateURL must be an Amazon S3 URL" seems another kind of problem. Did you copy the code with: aws s3 cp ./dist s3://$DEPLOY_BUCKET-$AWS_REGION/data-lake/$VERSION_CODE --recursive --acl bucket-owner-full-control including $VERSION_CODE in the end? Because this "detail" is missing in the Readme.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/awslabs/aws-data-lake-solution/issues/26?email_source=notifications&email_token=ALOKCOMDUAMLRCA34SMC64TQSLU2DA5CNFSM4G3NZT42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDHDDWY#issuecomment-550384091, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALOKCOOSCRS3NVXTIALJML3QSLU2DANCNFSM4G3NZT4Q .

sardbaba commented 5 years ago