search

vincentclaes / datajob

Build and deploy a serverless data pipeline on AWS with no effort.
https://pypi.org/project/datajob/
Apache License 2.0
110 stars 19 forks source link

example deploy fails `Error: Invalid S3 bucket name (value: data-pipeline-simple-None-deployment-bucket)` #49

Open petervandenabeele opened 3 years ago

petervandenabeele commented 3 years ago

The None has a capital letter which is invalid.

I ran:

export AWS_DEFAULT_ACCOUNT=_____________29
export AWS_PROFILE=my-profile
export AWS_DEFAULT_REGION=your-region # e.g. eu-west-1

<..>/datajob/examples/data_pipeline_simple$ datajob deploy --config datajob_stack.py 
cdk command: cdk deploy --app  "python <..>/datajob/examples/data_pipeline_simple/datajob_stack.py"  -c stage=None
jsii.errors.JavaScriptError: 
  Error: Invalid S3 bucket name (value: data-pipeline-simple-None-deployment-bucket)
  Bucket name must only contain lowercase characters and the symbols, period (.) and dash (-) (offset: 21)
vincentclaes commented 3 years ago

The None has a capital letter which is invalid.

I ran:

export AWS_DEFAULT_ACCOUNT=_____________29
export AWS_PROFILE=my-profile
export AWS_DEFAULT_REGION=your-region # e.g. eu-west-1

<..>/datajob/examples/data_pipeline_simple$ datajob deploy --config datajob_stack.py 
cdk command: cdk deploy --app  "python <..>/datajob/examples/data_pipeline_simple/datajob_stack.py"  -c stage=None
jsii.errors.JavaScriptError: 
  Error: Invalid S3 bucket name (value: data-pipeline-simple-None-deployment-bucket)
  Bucket name must only contain lowercase characters and the symbols, period (.) and dash (-) (offset: 21)

thanks Peter, I just noticed it today that this crashes with my latest changes. if you explicitly pass the stage argument it should work.

datajob deploy --config datajob_stack.py --stage dev

i plan to fix this tomorrow evening.

petervandenabeele commented 3 years ago

CloudFormation was successfully deployed. Now trying to run it ...

vincentclaes commented 3 years ago

fyi:

datajob execute --state-machine <your state machine name

to trigger the pipeline should work now

petervandenabeele commented 3 years ago

As mentioned in the README, I clicked "Start Execution" in the GUI and this has run successfully :-)

In the logs of the 3 AWS Glue (1.0) Jobs, I do see the "hello world" message, nice

petervandenabeele commented 3 years ago

The "destroy" (of the CloudFormation stack and IAM Role) also worked successfully, including the destruction of the CustomCDKBucketDeployment....

Only the CloudWatch log groups are still present. That might be OK.

Not sure if a finite Retention of 3 months or so would be better ?

/aws/lambda/data-pipeline-simple-dev-CustomCDKBucketDeployment-5N8SZ8C1NN9 | Never expire | - | - | -
-- | -- | -- | -- | --
  | /aws/lambda/data-pipeline-simple-dev-deployment-bucketBackend | Never expire | - | - | -
  | /aws/lambda/data-pipeline-simple-devBackend | Never expire | - | - | -

I will stop typing in this Issue.

I suggest to close this issue after either the README is fixed with the --stage dev explanation or some stage (e.g. dev) is set as default.

vincentclaes commented 3 years ago

good point, thanks Peter. we should create the cloudwatch logs as part of the stack

petervandenabeele commented 3 years ago

Before you do that (create snd destroy CloudWatch logs as part of CF (CloudFormation) stack): what is the life time of this CF stack? Would you want to see the Glue Job logs longer than the lifetime of this CF stack?