Create new feature for running and logging dbt commands

move-coop / parsons

A python library of connectors for the progressive community.

https://www.parsonsproject.org/

Other

259 stars 131 forks source link

Create new feature for running and logging dbt commands #841

Closed austinweisgrau closed 7 months ago

austinweisgrau commented 1 year ago

dbt is an increasingly popular tool in the data analytics community, for very good reason!

dbt's primary interface is through shell commands, which makes interacting with dbt commands programmatically somewhat difficult.

This new feature enables executing dbt commands from python, logs results to the python logger and optionally can also send formatted results to slack. The slack formatted message follows the formatting convention that dbt cloud uses for its slack messages.

austinweisgrau commented 1 year ago

Recent updates to dbt enable direct execution of dbt commands in python without needing to open a shell process. Using that dbt feature would be an improvement to this method. Docs on the new dbt feature here.

shaunagm commented 1 year ago

This looks great @austinweisgrau! Although I'm unfamiliar with dbt myself and hoping someone else will be able to also review this.

Some more general questions/thoughts:

Will this only work with Redshift? If so, is that because that's all you have time/access to build around, or is there a more fundamental reason (like the underlying functionality is build for Redshift)?
I wonder where the best place to put this is? Maybe in databases? I think not at the top level like it currently is.
We'll need docs and tests for these, before they can be merged.

Thank you again!

austinweisgrau commented 1 year ago

Good questions!

As currently built, will only work with Redshift. Should be simple to build modifications to enable it to work with other databases, but that's just what I am using and took the time to write up.
Good question! I don't have a great understanding of the organization of the package so I'm not sure.
Where's the best place to put the docs? Docstrings? Top of the module? In a readme?

austinweisgrau commented 1 year ago

Ok I finally looped back and made some updates:

Added instructions for how to install & use the utility
Moved the code from being its own connector to the utilities folder
Enabled installation with all the database engines that dbt itself supports

shaunagm commented 11 months ago

I'm planning on re-reviewing this in a few weeks, after we've done the DBT Fundamentals cohort and I have a better mental sense of what DBT is and how it fits with Parsons. Just wanted to update this PR so no one reads it and wonders why it's been sitting un-merged for over a month.

shaunagm commented 7 months ago

Hi @austinweisgrau - sorry I'm just now getting back to this. This still looks good to me, and after taking the dbt course I better understand the value of this utility. I'm going to update the branch and merge it.