Open lprzychodzien opened 2 months ago
Man you did tons of work here. Great job. Fundamental question themes:
Another major thing I will look at is taking a 50,000 foot view of the entire RxNorm data model - it's hard to see when zoomed in on a given intermediate model, but I think I re-wrote a lot of the code in other intermediate models inside of intermediate models. So I think there's an opportunity to ref intermediate models instead of re-writing that SQL.
Resolves #270
Explanation
Changes rxnorm staging and intermediate queries to using dbt's jinja table references.
Most important is that it sets intermediate models to materialize as tables (vs views).
Rationale
dbt mart models are made up of complex logic that should be captured in the intermediate models. The issues already that we ran into (issue #270 ) was that performing these aggregations for each query took 10+ minutes. Therefore, bringing this aggregation into a separated intermediate model to be materialized as a table is a good solution. The materialization of these intermediate tables can take awhile but will significantly speed up queries.
Additional work can be done to optimize the queries that build out intermediate models in the future.
Tests
testing logs
``` ```