eric-gt
commented
3 years ago Data Pathway Requirements
General
Anything reported from Firestore to DataStudio will likely follow the following pipeline: Firestore collection >>CloudFunction Event Trigger>> BigQuery table >>Data Studio Connector>> DataStudio Report
Where possible, sanity checks will avoid firestore entirely. If data is not available outside firestore (i.e: assignment data, open donations) estimated trends can be plotted based on expected behavior and base metrics (e.g. 10 donations opened on 2021-03-01, 9 for $20, 1 for $50 at $1/learner over 14 days, on 2021-03-13 expected open donations should be 9 at 18 learners, 1 at 47 learners)
Business Metrics
Learners Acquired
first_openevents- per app
- per country
- per referral source (specific ad campaigns, partner referral links, etc.)
- Reported from Firestore with sanity check scraped from individual Analytics buckets in BigQuery
- % difference from yesterday's ingestion
Donors Acquired
- new donor documents
- by referral source (facebook, twitter etc.)
- by FtL campaign
- reported from Firestore with sanity check from Stripe
- % difference from previous day
Donations in USD
- total donation amounts
- per donor
- monthly or one-time
- by referral source (facebook, twitter etc.)
- by FtL campaign
- reported from Firestore with sanity check from Stripe
- % difference from yesterday's amounts
Tech Health Metrics
Learners Ingested
see Business Metrics >> Learners Acquired
Active Donations
- donation documents without an end date
- by percent filled
- by start date
- highlight donations older than a week and donations with no learners
- Reported from Firestore with calculated estimates line
Learners Assigned
- learners with
assignedOnfield- by date
- by country
- by FtL campaign
- Reported from Firestore with calculated estimates line
Status Deep Dives
Postman
- number of healthy endpoints
- errors in past hour
Stripe
- number of successful payment intents in last 24 hours
- number of errors in last 24 hours
Cloud Functions
- Avg function response time
- Avg function execution time
- errors in last hour
Firestore
- Daily Reads
- Daily Writes
- Daily Deletes
EC2-instance
- server error logs
Ingestion Cronjobs
- number of users created in last 24 hours
- number of users assigned in last 24 hours
- error log
User Story As a developer, I want to review the various data pathways for obtaining the data needed to power the three sections of the Admin Dashboard (Business Overview, Tech Health, and Deep Dive Tech), So that I can determine what best practices for how/when we should pull data to employ and what backend infrastructure would be needed to store data (if needed).
As a developer, I want to create any backend data structures necessary to hold data over time to power the 3 Admin Dashboard sections, So that the visualizations can be powered off the
Acceptance Criteria Given that I am viewing the Admin Dashboard, When I browse to either the Business Overview section, the Tech Health section, or the Deep Dive Tech section, Then I should be fetching data necessary to power the visualizations in each of those sections.