medic / cht-pipeline

CHT Pipeline is a tool used to define data models for transforming the raw data we get from Couch DB into models that can then be queried to build dashboards
GNU General Public License v3.0
1 stars 4 forks source link

A set of SQL queries that transform raw CouchDB data into a more useful format. It uses dbt to define the models that are translated into PostgreSQL tables or views, which makes it easier to query the data in the analytics platform of choice.

Local Setup

Follow the instructions in the Local CHT Sync Setup documentation to set up CHT Sync locally.

Run dbt models unit tests locally

Prerequisites

Run the tests

  1. Navigate to tests folder.
  2. Run the test script
# set environment variables, install dbt dependencies, seed data, run dbt, run test
./run_dbt_tests.sh

Release Process

This repo has an automated release process where each feature/bug fix will be released immediately after it is merged to main. The release type is determined by the commit message format. Have a look at the development workflow in the Contributor Handbook for more information.

Commit message format

The commit format should follow the convention outlined in the CHT docs. Examples are provided below.

Type Example commit message Release type
Bug fixes fix(#123): rename column names patch
Performance perf(#789): add new indexes patch
Features feat(#456): add new model minor
Non-code chore(#123): update README none
Breaking perf(#2): remove data_record model
BREAKING CHANGE: form models should now read from new_model
major