Automate The ETL Pipeline Status Check Up

MacHu-GWU commented 5 years ago

User story

As the analytics ETL pipeline developer, I would like to automate the status / data healthy checkup, so that I don't have to log in to AWS Console / jumphost to manually run some script over and over again.

The manual check up:

SSH to Jumphost EC2 instance, manually run some query to identify number of new ingested data.
Login to AWS Console, visually check some invocations with errors, and go to Cloudwatch to identify whether it is normal (Time Consuming).

Notes

What is the value to the user in this story?

The business persons are periodically checking the QuickSight dashboard. They found the Pipeline Outage on Sep 25, 2018 and notified analytics team.

Since there is no user periodically checking QuickSight right now, and there's no easy way to allow non developer checking the ETL pipeline status. It is really hard to know that if there's an outage when @MacHu-GWU is out.

What are things we should consider when making this story

[ ] the automation solution is not decided yet
[ ] the automation also obeys the compliance rules
[ ] amount Effort to take

Acceptance Criteria

[ ] Non-developer should be able to check up the ETL Pipeline status and knows who to reach out for help.

Tasks to complete the story

[ ] Document for checkup, debug
[ ] Automate Script / System

Definition of Done

[ ] Pull requests meet technical definition of done
[ ] Usability tested

lauraGgit commented 5 years ago

@MacHu-GWU it seems like we should wait until the new pipeline is implemented. What do you think?

MacHu-GWU commented 5 years ago

@lauraGgit

There are two parts to check for Pipeline Status:

Check Redshift Data
Check Pipeline Invoke Stats, duration, error rate.

I think <1> will not change in the new solution. We can still do <1>.

lauraGgit commented 5 years ago

migrated to jira

18F / identity-analytics-etl