[ADAP-580] [Feature] Query outcome logging

matsavage commented 1 year ago

Is this your first time submitting a feature request?

[X] I have read the expectations for open source contributors
[X] I have searched the existing issues, and I could not find an existing issue for this feature
[X] I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion

Describe the feature

I would like a logging line post SQL execution to inform me how many rows were affected/inserted/updated/deleted if modifications are made to a materialised table.

Describe alternatives you've considered

Toyed with the idea of using a post-hook to investigate the output of the previous query, however this seems

Who will this benefit?

This will help in a number of situations, particularly one I am currently facing, where I am trying to uptimise loads into our data warehouse where I am trying to lower the number of rows we update when we do a database refresh. This data will allow me to track the efficacy of the solutions I am trying to achieve this goal.

Will also help for alerting, eg, we expect table x to grow by n rows per day, can we monitor that.

Are you interested in contributing this feature?

Possibly

Anything else?

No response

Fleid commented 1 year ago

Hey @matsavage - thanks a lot for opening this issue!

The topic has already been explored here: https://github.com/dbt-labs/dbt-spark/issues/497 Would you mind taking a look at the conversation and see if you'd like to pick it up? We won't be able to do so on our end.

matsavage commented 1 year ago

Thanks for the update, have replied in the other thread, will close this one

dbt-labs / dbt-spark