dbt-labs / dbt-redshift

dbt-redshift contains all of the code enabling dbt to work with Amazon Redshift
https://getdbt.com
Apache License 2.0
96 stars 59 forks source link

[Feature] Provide `query_id` in the Adapter Response #892

Open KeltonKarboviak opened 2 months ago

KeltonKarboviak commented 2 months ago

Is this your first time submitting a feature request?

Describe the feature

I want to be able to capture the query ID of the statement used to load a model.

This will be helpful if we ever need to troubleshoot or diagnose a particular model load by taking the query ID and cross-referencing it in the various system tables that give stats & metrics for every query that is run.

Describe alternatives you've considered

I am unsure of an alternative for how to capture the exact query ID that was run for a model besides capturing it in the adapter and providing it in the Adapter Response, which will then save it to the JSON artifacts.

Who will this benefit?

This will be helpful for database administrators, data engineers, and data analysts in needing to troubleshoot or diagnose performance issues with a particular dbt build.

Are you interested in contributing this feature?

Yes

Anything else?

Reference to the dbt-snowflake adapter where they are capturing query_id in the adapter response: https://github.com/dbt-labs/dbt-snowflake/blob/f95b9192f6eec9af4e30eaab87f9e3412febf7d1/dbt/adapters/snowflake/connections.py#L456-L461

amychen1776 commented 1 month ago

@KeltonKarboviak Thank you for opening this (and the draft PR)! One of the concerns I have with your draft PR is the performance impact for accessing another metadata table as well as the fact that AWS has been recommending that we migrate dependencies off of the pg tables for a bit now. Would you be able to check for performance impact in the PR?

I'm curious of what you think of the alternative approach of tagging query history would also meet your needs?