Closed metadimensions closed 2 years ago
I think you can. What you need to do is organizing glue jobs, triggers, crawlers in the workflow and attach the resources such as s3 to the crawler and db in the template.
It is possible to provision Glue workflow (maybe you wanted to mention Glue workflow in "Glue pipeline") via multiple different options.
For example, this blog post has an example to provision Glue workflow via CloudFormation. https://aws.amazon.com/blogs/big-data/build-a-serverless-event-driven-workflow-with-aws-glue-and-amazon-eventbridge/
On the other hand, this blog post has an example to provision Glue workflow via Glue custom blueprint. https://aws.amazon.com/blogs/big-data/simplify-data-integration-pipeline-development-using-aws-glue-custom-blueprints/
BTW, this repo is mainly for AWS Glue samples. Please ask general questions in AWS Forum or AWS Premium Support.
Hey @moomindani
Thanks for the reply.
I tried the pipeline using Cloudformations but now I have a doubt on how to update the rows in RDS DB - Postgres using the same AWS Glue
For eg: If the CSV files from S3 Bucket has some rows updated so how can we update the same into the RDS DB because I tried running the job but it shows some duplication error.
Hello,
I have created a pipeline manually using AWS Glue to import files from S3 to Aurora DB
Is there a way we can do this same via Cloud formations?
Thanks in advance.