search

ActoKids / AD440_W19_CloudPracticum

3 stars 1 forks source link

Outline and investigate triggers #19

Open mrvirus9898 opened 5 years ago

mrvirus9898 commented 5 years ago

Our crawlers will need triggers to know when to execute their code. Please investigate how AWS handles triggers for their Lambda functions, and describe the options to us. If AWS has a GUI for handling triggers, please take screen shots for us.

Time permitting, please instance a very simple script that pings (or whatever positive notification is easiest) a target based off a trigger.

Please indicate the time spent on this, any issues that you are having, any good references you found for this subject, and credit anyone helped you out.

mrvirus9898 commented 5 years ago

Will assign to Tyler Reed when he joins this github

TyReed12 commented 5 years ago

AWS Glue offers three types of triggers:

Schedule A time-based trigger based on cron.

Job events (conditional) An event-based trigger that fires when a previous job or multiple jobs satisfy a list of conditions. You provide a list of job events to watch for when their run state changes to succeeded, failed, stopped, or timeout. This trigger waits to fire until any or all the conditions are satisfied.

On-demand The trigger fires when you start it. As jobs complete, any triggers watching for completion are also fired and dependent jobs are started.

So that they are ready to fire as soon as they exist, you can set a flag to enable (activate) schedule and job events (conditional) triggers when they are created.

TyReed12 commented 5 years ago

AWS Glue Docs

TyReed12 commented 5 years ago

Nick, in order to test a trigger I will need access to the AWS Glue console.

TyReed12 commented 5 years ago

I spent three hours researching glue and their different features. The only issues I had were that the docs are extensive and glue offers way more features than we need. But the AWS Glue docs are very well organized and relatively easy to understand.

TyReed12 commented 5 years ago

Deliverable: Using Glue, set a scheduled trigger to call John's API

TyReed12 commented 5 years ago

Update: I've spent approximately 6 hours today looking at ways of calling John's API. I've created my own AWS account and added S3, Lambda, and Glue services. The problem I'm having currently is configuring the connection options for the job that needs to trigger. It's giving me the choice of three connection types: Amazon Redshift, Amazon RDS, and JDBC.

After further digging I was able to find that AWS Glue offers support for Boto3. I've been in contact with @MrSamsa42 over slack and have asked for a code snippet of his lambda function.

mrvirus9898 commented 5 years ago

Rodger that. Hopefully Glue and Lambda don't give us much trouble.

TyReed12 commented 5 years ago

Received further clarification from Toddy. He asked whether there was another AWS service could be used to schedule triggers for Lambda functions. Going to try to get through this tutorial today for CloudWatch: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html

TyReed12 commented 5 years ago

Was able to complete tutorial with a test function and setup a rule in CloudWatch that triggers the lambda function once per day. Will need to discuss with team where to move forward from here.

TyReed12 commented 5 years ago

Estimated time to be spent: 3 hours Approximate time spent: 10 hours

After much digging and research, it appears Glue seems to be built to only connect to databases/crawlers. When it comes to triggering lambda functions, there are many AWS services that can handle those triggers. I used CloudWatch to make a once per day scheduled trigger for a test lambda function and it was relatively simple.