search

medic / cht-sync

Data synchronization between CouchDB and PostgreSQL for the purpose of analytics.
GNU General Public License v3.0
4 stars 5 forks source link

Allow DBT MODELS to be specified from file location #182

Open 1yuv opened 2 weeks ago

1yuv commented 2 weeks ago

Is your feature request related to a problem? Please describe. cht-sync currently requires CHT_PIPELINE_BRANCH_URL to be set and requires all dbt models to be committed on github.

Describe the solution you'd like Current practice makes is difficult to develop or test any new models as it requires the new models need to be pushed on github without testing. Enable alternative environment variable DBT_PIPELINE_PATH that can specify relative or absolute path to pipeline models.

Describe alternatives you've considered None

witash commented 1 week ago

you can use the e2e test framework for local testing, which is what we did for developing the base models. e2e tests are designed for a local couchdb with fake data, but can use a remote couchdb also.

edit models at tests/dbt/package copy tests/.e2e-env to tests/.local-test-env and change couchdb url run with

docker compose --env-file ./tests/.local-test-env -f docker-compose.yml -f docker-compose.postgres.yml -f tests/dbt/docker-compose.yml up -d --build

but this is not documented and easy to use.

we can add something to the dbt dockerfile so it copies files into the container...or keep it this way (where it uses DBT_TARBALL_URL env variable, which points to another container that's built by tests/dbt/docker-compose.yml) but just clean up the commands and add it to the docs

mrjones-plip commented 1 week ago

😍

clean up the commands and add it to the docs