OHDSI / dbt-synthea

[Under development] A dbt ETL project to convert a Synthea synthetic data set into the OMOP CDM
https://ohdsi.github.io/dbt-synthea/
Apache License 2.0
16 stars 6 forks source link

Add a "dev mode" variable to toggle on materialization of stg and int models #38

Open katy-sadowski opened 5 months ago

katy-sadowski commented 5 months ago

Currently, all stg and int models are materialized in the database. This clutters the schema and is not desirable for production analytic use cases. However, developers (and analysts in some cases) may want/need to inspect the contents of these models.

We can use a dbt_project variable that users pass into the command line that will enable materialization of stg and int models (I think it should be disabled by default).

lawrenceadams commented 1 month ago

I've tried this before and the solution that we used was to instead have a post run hook or a macro that drops all objects except the core tables we want i.e. the final OMOP tables.

At the moment our final dbt_synthea_dev schema holds 85 tables. We could rationalize those to be views instead? This may have a performance hit downstream.

Another approach would be to have our OMOP models be in their own schema - but I am unsure if that's really better // its just more complex

katy-sadowski commented 1 month ago

The post-run hook to drop all but final OMOP models sounds like a great idea to me!