fvankrieken
commented
4 months ago So there are a few parts to this.
- [x] simplify staging models. I think this will be pretty minimal, but we should glance over and see if there are simple changes to make - do two consecutive CTEs do very little differently, and could we combine them? Stuff along those lines
- [x] simplify intermediate models other than
int__zoningtaxlots. This has a few parts- move logic from "dcp_zoningtaxlots" table upstream. There's a bunch of logic relating to some final filtering, altering etc that ideally could happen upstream from this final join table.
- Simplify/clean existing code in intermediate models. Evaluate final columns selected - are these used downstream? Are these helpful as persisted intermediate columns for debugging?
- Edit constraints/tests. Ideally we want to be as constraining as possible here (unique set of columns ideally is just 2 columns)
- simplify query that generates
int__zoningtaxlots. We can flush this out after we get upstream logic ironed out
HengJiang0206
After we have a working dag/dbt project from #925, it'd be good to evaluate our DAG/sql and iterate on our models. Did we compress too much and now have queries that are too busy and hard to read? Or are some models maybe too discrete, and could be grouped further? This can be a bit of a gray area, but to me it comes down to largely
There's also likely a macro to be written. The code for all the "per area" files/models is very repetitive and likely could have some of it pulled out