search

DostEducation / whatsapp-webhook-analytics

Handling and processing of Incoming webhook request configured at Glific.
GNU Affero General Public License v3.0
0 stars 0 forks source link

Documentation of Airbyte based infrastructure #41

Closed Satendra-SR closed 3 months ago

Satendra-SR commented 3 months ago

TODO

Need to create documentation for AirByte-based infrastructure setup for Glific Webhook database migration

Satendra-SR commented 3 months ago

Documented on Notion - https://www.notion.so/dosteducation/Airbyte-WhatsApp-Webook-database-Postgres-to-BigQuery-Migration-54cb5bef5bae48a7a40008e673881d58?pvs=4

Satendra-SR commented 2 months ago

Sharing a breakdown of this item for future reference:

Initial estimated scope - 4 hours

  1. High-level overview of how things are configured.
    • Data Extraction
    • Transformation (optional)
    • Staging in GCS
    • Loading into BigQuery
  2. All the components involved in the integration
    • Application Database
    • Airbyte
    • Bigquery
    • GCS staging
  3. Key Considerations
  4. Benefits of Using GCS Staging over standard Inserts (direct insert to BigQuery)
  5. Flow diagram

Based on the team discussion, the updated scope of work - 6 hours

  1. Step-by-step Setup Guidelines - to setup similar pipeline in future
  2. Steps to change replication frequency - scheduler frequency
  3. Steps to change the sync mode - to make sure anyone can change the type of replication (overwrite, append, etc)