search

18F / identity-analytics-etl

ETL and SQL scripts for Login.gov data warehouse and business intelligence
10 stars 6 forks source link

Task: user flow diagram #180

Closed lauraGgit closed 5 years ago

lauraGgit commented 5 years ago

User story

As a new member of the analytics team, I would like to see a diagram of the data pipeline, so that I can understand the implications of the data quality and breath.

Notes

Tasks to complete the story

Definition of Done

MacHu-GWU commented 5 years ago

@lauraGgit I can draw it when I finished task in this Sprint

lauraGgit commented 5 years ago

Draft user event data-pipeline diagram

image

Mermaid Chart to generate this:

graph LR
A[Production Application] -->|User driven event| B[AWS CloudWatch]
A -. User driven event.-> L[Google Analytics]
B --> C[Lambda Parser]
C --> G[Intermediate S3 bucket]
D --> I[Lamdba -RedShift Loader]
I --> J[RedShift]
J --> H[AWS QuickSite]
J -.-> K[Blazer]
B --> D[Logstash]
D --> E[ElastiSearch]
E --> F[Kibana]
lauraGgit commented 5 years ago

Update per Sanhe's comments:

image

Live Link

Code to generate

graph LR
A[Production Application] -->|User driven event| B[AWS CloudWatch]
A -. User driven event.-> L[Google Analytics]
D --> M[Raw data s3 bucket]
M --> C[Lambda Parser]
C --> G[Hot bucket -Intermediate S3 bucket]
G --> I[Lamdba -RedShift Loader]
I --> J[RedShift]
J --> H[AWS QuickSite]
J -.-> K[Blazer]
B --> D[Logstash]
D --> E[ElastiSearch]
E --> F[Kibana]
lauraGgit commented 5 years ago

Now with Kinesis future implementation: image

graph TD
A[Production Application] -->N[User driven event]
N -.-> L[Google Analytics]
N --> B[AWS CloudWatch]
B -.-> O[Kinesis]
O-.->J
D --> M[Raw data s3 bucket]
M --> C[Lambda Parser]
C --> G[Hot bucket -Intermediate S3 bucket]
G --> I[Lamdba -RedShift Loader]
I --> J[RedShift]
J --> H[AWS QuickSite]
J -.-> K[Blazer]
N --> D[Logstash]
D --> E[ElastiSearch]
E --> F[Kibana]
MacHu-GWU commented 5 years ago

@lauraGgit I think we can mark this one https://github.com/18F/identity-analytics-etl/issues/180#issuecomment-461181600, as out-of-dated.

MacHu-GWU commented 5 years ago

And I think we can mark this as done