search

aws-samples / aws-cdk-examples

Example projects using the AWS CDK
Apache License 2.0
5.12k stars 2.15k forks source link

CDK Example for AWS Glue Workflow including Glue Jobs, Triggers, Crawlers, securtiy group and Database #438

Open PuneetBabbar opened 3 years ago

PuneetBabbar commented 3 years ago

:rocket: Feature Request

General Information

Description

This example will be like another example in the REPO. This new feature request will add another example and will cover everything around AWS Glue, containing AWS Glue workflow, Glue Jobs, Triggers, Crawlers, security group and Database.

This will enhance and add new feature. As currently there is no code example available online and also no AWS documentation with code example to demonstrate on how can can use CDK to setup an ecosystem around Glue.

Proposed Solution

The idea is to build an example that will setup an ETL pipeline using tool set available in Glue ecosystem, and implement or construct it via complete CDK code.

I was thinking to open source data and build an example for a Data ETL job, something similar to blog like https://aws.amazon.com/blogs/devops/provision-codepipeline-glue-workflows/ or something similar. But the idea is to build an ETL pipeline via workflow. And have steps constructed via CDK

  1. Glue Crawler to catalog S3 data.
  2. Glue Jobs (Spark) to process and transform the catalog data
  3. Glue Trigger for calling the above Crawler and Jobs
  4. Glue Workflow to orchestrate the above components.

I don't have the diagram for the workflow, but can create and will be able to add to the example for better understanding.

Environment

Other information

PuneetBabbar commented 3 years ago

@NGL321 can this be assigned to me ? So can start working on this. thanks

NGL321 commented 3 years ago

We assign internally to the team for who is supposed to shepherd the process. But please feel free to take this on yourself!

Quick heads-up that we are backed up on reviewing contributions to this repo due to some considerations about how to grow/develop it, but if you contribute a PR we will review it as able!

For my part, I think a well-written Glue example would be very useful.

😸 😷

maddy2u commented 3 years ago

@PuneetBabbar - Are you coming up with this? Would be very useful